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ABSTRACT 


This  thesis  investigates  the  feasibility  of  applying  a  knowledge-based 
approach  to  the  problem  of  estimating  the  future  success  of  a  major  defense 
acquisition.  This  thesis  will  model  the  US  Government  Accountability  Office  (US 
GAO)  knowledge  based  methodology  for  evaluating  programs.  This  methodology 
relies  on  three  sets  of  knowledge,  namely,  technology,  design,  and  production.  In 
particular,  the  technology  dimension  is  measured  by  Technology  Readiness 
Level  (TRL),  as  defined  by  NASA.  In  addition,  the  methodology  relies  on 
assessing  the  design  readiness  of  a  program  by  examining  the  status  of  the 
release  of  engineering  drawings.  Finally,  the  US  GAO  methodology  assesses 
the  production  readiness  of  a  program  by  examining  the  status  of  Statistical 
Process  Quality  Control  (SPQC)  procedures.  This  thesis  also  presents  to  the 
ROKAF  decision  maker  an  independent,  knowledge-based  estimation  of  the 
ROKAF  T-50  program  by  applying  the  US  GAO  methodology  to  the  T-50 
program. 
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EXECUTIVE  SUMMARY 


This  thesis  investigates  the  knowledge-based  approach  developed  by  the 
US  GAO  to  estimate  if  the  weapon  acquisition  program  could  meet  requirements 
without  cost  overruns  and  schedule  delays.  To  manage  the  major  weapon 
acquisition  program  and  achieve  the  key  results,  the  US  GAO  has  developed 
three  knowledge  points;  technology  maturity,  design  stability  and  controlled 
production  stability.  The  T-50  acquisition  program,  the  newly  developed  training 
aircraft  of  ROK  Air  Force,  is  subjected  to  an  assessment  by  the  GAO 
methodology  to  ascertain  if  the  program  would  enjoy  good  progress  in  the 
research  and  development  phase.  The  10  US  programs’  data  are  also  compared 
with  the  T-50  program  to  evaluate  the  technology  maturity  of  those  programs  and 
judge  if  they  are  following  GAO  guidance. 

The  following  shows  the  status  of  whether  each  program  in  our  dataset 
follows  the  GAO  guidance  or  not.  GAO  considers  each  of  these  as  a  “best 
practice”  in  the  major  weapon  acquisition  process 

•  Technology  Readiness  Level(TRL)  above  7 

•  90  percent  achievement  of  engineering  drawings  at  Critical  Design 
Review  (CDR) 

•  Production  supported  by  Statistical  Process  Control  (SPC) 

The  GAO  knowledge  based-methodology  about  T-50  program  finds  some 
conclusions  with  the  above  rules. 

•  Since  the  T-50  program  achieved  a  TRL  of  6.3  at  the  development 
stage,  we  believe  that  the  T-50  program  will  achieve  the  goal  of 
TRL  7  with  the  trend  of  technology  development. 

•  The  releasable  drawings  of  T-50  program  at  CDR  is  not  at  a 
sufficient  level  to  ensure  design  stability.  T-50  has  more  design 
drawings  than  the  advanced  programs  in  our  dataset  as  well  as  the 
unmanned  programs  in  our  dataset.  The  T-50  program’s 
achievement  of  design  drawing,  67%,  at  CDR  is  below  the  “best 
practice”  guidance  of  the  GAO. 
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T-50  prototype  production  had  been  performed  without  fully 
developed  statistical  control  and  production  is  now  at  the  beginning 
phase  of  mass  production.  However,  there  has  not  been  an 
instance  where  SPC  has  been  implemented  directly  for  each 
component  of  the  T-50  prototype  model.  On  the  other  hand,  the  T- 
50  program  has  a  specific  plan  of  SPC  for  implementation  during 
the  Full  Scale  Development  and  Production  phases. 


I.  INTRODUCTION 


A.  BACKGROUND 

The  Republic  of  Korea  Air  Force  (ROKAF)  Center  has  been  studying 
military-force  increments  within  the  context  of  constraints  imposed  by  shortfalls  in 
the  ROKAF  budget  allocations.  The  larger  context  is  that  the  Korean 
governmental  budgetary  deficit  acts  as  a  severe  constraint  on  the  general  ability 
to  fund  both  ROK  defense  and  improvements  to  ROKAF.  Specifically,  both  new 
and  ongoing  weapons-acquisition  programs  are  under  financial  pressure.  In 
response,  ROKAF  has  encouraged  efficient  cost  evaluation,  aiming  to  employ  a 
limited  budget  wisely. 

Some  weapons-acquisition  programs  already  in  progress  have  failed  to 
arrive  at  stated  goals,  compromising  related  programs  in  turn.  The  chronic 
problem  of  delays  and  overruns  has  boxed  the  ROKAF  into  attempting  the 
development  and  fielding  of  ever-more-advanced  capabilities,  subject  to 
increasingly  rigid  cost  and  time  constraints.  Despite  these  budgetary  pressures, 
the  master  plan  for  defense-weapons  acquisition  can  be  expected  to  meet 
challenges  such  as  cost  increases  and  scheduling  delays  if  cost  evaluation  is 
included  at  the  beginning  of  each  component  program  and  as  an  ongoing  part  of 
the  analysis  of  each  program.  Before  approval  is  given  for  final  investment  in 
the  Korean  master  weapon-acquisition  program,  a  detailed  cost  estimation 
should  be  in  place. 

As  a  case  in  point,  the  ROKAF  has  invested  in  the  development  of  the  T- 
50  training  aircraft  from  concept  through  development,  beginning  at  the  end  of 
1992.  The  T-50,  a  next-generation  training  aircraft  built  by  Korea  Aerospace 
Industries  (KAI)  and  Lockheed  Martin  Aeronautics  Company,  features  the  power, 
performance,  avionics,  and  cockpit  display  of  the  most  sophisticated  defense 
systems.  Using  the  production  of  this  aircraft  to  illustrate  the  value  of  cost  and 
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schedule  estimation  will  demonstrate  to  ROKAF  headquarters  the  importance  of 
evaluating  a  weapon-acquisition  program  and  offer  insights  towards  application 
to  other  programs. 

B.  OBJECTIVE 

The  objective  of  this  thesis  is  to  investigate  the  feasibility  of  applying  a 
knowledge-based  approach  to  the  evaluation  of  a  major  defense-acquisition 
program-  that  of  the  ROKAF  T-50  aircraft  and  to  investigate  options  for  adapting 
the  US  GAO  methodology  to  the  ROKAF  T-50  aircraft  program.  Specifically,  this 
thesis  demonstrates  how  a  knowledge-based  approach  to  the  problems  of  the 
master  defense-weapon  acquisition  program  can  be  defined  and  employed,  and 
offers  practical  risk-assessment  tools  to  weapons-acquisition  officers. 

C.  RESEARCH  QUESTIONS 

This  thesis  will  investigate  the  following  questions: 

•  Can  U.S.  GAO  methodology  be  applied  to  the  ROKAF  T-50 
program? 

•  What  are  the  differences  between  the  U.S.  and  ROKAF  programs, 
as  measured  by  GAO  methodology? 

•  What  can  we  learn  about  the  forecast  of  success  and  cost  overruns 
in  ROKAF  programs? 

D.  SCOPE  AND  LITERATURE  REVIEW 

1.  Scope 

This  thesis  examines  the  prospects  of  success  and  the  risk  of  cost 
overruns  in  a  ROKAF  T-50  aircraft  development-and-procurement  program. 

There  are  two  possible,  and  different,  approaches  to  examining  forecasted 
costs  associated  with  a  weapons-acquisition  program: 

•  Focus  on  the  affordability  of  system  development,  using  the  work  of 
the  advanced  cost-estimating  integrated-product  team  (IPT),  as 
was  done  in  the  Joint  Strike-Fighter  (JSF)  program.  The  IPT  is 
then  tasked  with  and  responsible  for  developing  methods  to  assess 
program  initiatives. 

•  Use  GAO  methodology  to  assess  technology  maturity  based  on 
three  knowledge  points. 

Because  the  GAO’s  knowledge-based  methodology  provides  the  greater 
insight  into  risks  of  cost  overrun,  this  thesis  applies  the  GAO  approach. 
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Discovery  of  whether  a  knowledge-based  approach  can  be  applied  directly  to  a 
ROKAF  acquisition  case  will  ultimately  give  officers  in  charge  another  tool  to 
enhance  their  ability  to  plan  appropriately  for  projects  and  procurement. 
Developing  and  advancing  the  US  programs  will  be  used  in  comparison  with  the 
ROKAF  T-50  program. 

This  thesis  requires  the  use  of  T-50  program  data  and  assessment  of 
technical  maturity  on  the  basis  of  NASA  technology-readiness  levels  (TRLs)  It 
also  implements  the  latest  acquisition-report  information  available.  As  part  of  a 
data-quality  assurance-and-validation  program,  we  sought  data  whose  reliability 
was  based  upon 

•  Eliminating  date  that  was  anomalous  or  incredible 

•  Reliable  sources,  such  as  program  managers 

2.  Literature  Review 

The  US  GAO  has  written  extensively  on  ways  to  enhance  the  weapon 
acquisition  process.  For  example, 

Although  the  weapons  that  the  Department  of  Defense  (DOD) 
develops  have  no  rival  in  superiority,  there  still  remain  ways  in 
which  they  can  be  improved.  GAO’s  reviews  over  the  past  20  years 
have  found  consistent  problems  with  weapon  acquisitions-cost 
increases,  schedule  delays,  and  performance  shortfalls-along  with 
underlying  managers  to  promise  more  than  they  can  deliver.  DOD 
can  resolve  these  problems  by  using  a  knowledge-based  approach 
derived  from  the  best  practices  of  successful  product 
developments.1 

As  essential  parts  of  successful  program  management,  the  best  weapons- 
acquisition  programs  take  extra  steps  to  confirm  technology  maturity,  design 
stability,  and  production  stability. 

Separating  technology  development  from  product  development  is 
important  to  this  effort.  Successful  programs  make  a  science  and 
technology  organization,  rather  than  the  program  or  product 


1  GAO-04-248,  Assessments  of  Major  Weapon  Programs,  p.  2. 
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development  manager,  responsible  for  maturing  technologies.  Such 
steps  can  help  to  reduce  costs  and  deliver  a  product  on  time  and 
within  budget.2 

3.  Organization  and  Chapter  Summary 

This  section  gave  an  overview  of  the  thesis  and  the  process  of  assessing 
technical  maturity  based  on  a  GAO  report. 

•  Chapter  II  presents  information  necessary  to  understanding 
theoretical  cost  analysis,  the  GAO  methodology,  and  statistics. 

•  Chapter  III  looks  at  application  of  theory  and  compares  data 
pertaining  to  the  U.S.  and  T-50  master  acquisition  programs. 

•  Chapter  IV  provides  conclusions  and  recommendations  for  further 
research. 


2  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  6. 
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RELATED  RESEARCH 


A.  WEAPON  ACQUISITION  ENVIRONMENT 

1.  Introduction 

The  program  manager  also  should  consider  the  requirements  of  users  in 
various  combatant  services.  Though  weapon  acquisition  program  managers  and 
decision  makers  have  been  struggling  to  achieve  desirable  technical  levels  within 
schedule  and  budget,  the  acquisition  process  has  proved  somewhat 
unpredictable.  For  example, 

the  US  weapons-acquisition  program  results  in  reduced  quantities 
and  increased  costs.  The  JSF  acquisition  program’s  estimated 
development  and  procurement  costs  have  increased.  In  addition, 
the  number  of  aircraft  it  plans  to  deliver  has  been  reduced.  As  a 
result,  unit  costs  for  the  JSF  aircraft  have  increased  substantially, 
thereby  reducing  the  program’s  buying  power.  The  most  significant 
quantity  reduction  occurred  after  system  development  began  in 
2001,  when  the  program  reduced  the  number  of  aircraft  it  plans  to 
procure  from  2,852  to  2,443,  or  by  14  percent.  The  Navy-concerned 
that  it  could  not  afford  the  number  of  tactical  aircraft  it  planned  to 
purchase-reduced  the  number  of  JSF  aircraft  for  joint  Navy  and 
Marine  Corps  operations  from  1,089  to  680  by  reducing  the  number 
of  backup  aircraft  needed.3 

2.  The  T-50  Program 

The  T-50  is  a  true  digital  aircraft  with  the  power,  performance, 
avionics  and  cockpit  display  of  today's  most  sophisticated  defense 
systems.  Everything  about  the  T-50  readies  student  pilots  for  the 
rigors  of  frontline  fighters.  Through  the  joint  development  team  of 
Korea  Aerospace  Industries  and  Lockheed  Martin  Aeronautics 
Company,  an  all-new  trainer  bridges  the  gap  between  subsonic 
training  and  supersonic,  next-generation  fighters.  The  T-50  easily 
exceeds  Mach  1  and  offers  the  widest  flight  envelope  of  any 
advanced  jet  trainer  (AJT)  and  lead-in  fighter  trainer  (LIFT) 
available.  Pilots  take  the  controls  of  a  smooth-handling  jet  that 
offers  the  exceptional  maneuverability  of  next-generation  fighters, 
with  similarly  integrated  cockpit  and  systems.4 


3  GAO  05-271,  Joint  Strike  Fighter  Acquisition,  p.  7. 

4  ROK  Air  Force,  http://www.airforce.mil.kr/ENG/index.html  (Accessed  October  24,  2005) 
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3.  The  US  Acquisition  Programs 

This  thesis  will  use  the  US  programs  for  comparison  with  the  T-50 
program.  A  brief  description  to  understand  the  general  of  each  program  follows. 

•  B-2  Radar  Modernization  Program  (RMP):  “The  Air  Force’s  B-2 
RMP  is  designed  to  modify  the  current  radar  system  to  resolve 
potential  conflicts  in  frequency  band  usage.  To  comply  with  federal 
requirements,  the  frequency  must  be  changed  to  a  band  where  the 
B-2  will  be  designated  as  a  primary  user.  The  modified  radar 
system  is  being  designed  to  support  the  B-2  stealth  bomber  and  its 
combination  of  stealth,  range,  payload,  and  near  precision  weapons 
delivery  capabilities.”5 

•  C-130  Avionics  Modernization  Program  (AMP):  “The  Air  Force’s  C- 
130  AMP  standardizes  the  cockpit  configurations  and  avionics  for 
14  different  mission  designs  of  the  C-130  fleet.  It  consolidates  and 
installs  the  mandated  DOD  Navigation/Safety  modifications,  the 
Global  Air  Traffic  Management  systems,  and  the  C-130  broad  area 
review  requirements.  It  also  incorporates  other  reliability, 
maintainability,  and  sustainability  upgrades  and  provides  increased 
situational  awareness  capabilities  and  reduces  susceptibility  of 
Special  Operations  aircraft  to  detection/interception.”6 

•  C-5  Avionics  Modernization  Program  (AMP):  “The  Air  Force’s  C-5 
AMP  is  the  first  of  two  major  upgrades  for  the  C-5  to  improve  the 
mission  capability  rate,  transport  capabilities  and  reduce  ownership 
costs.  The  AMP  implements  Global  Air  Traffic  Management, 
navigation  and  safety  equipment,  modern  digital  equipment,  and  an 
all-weather  flight  control  system.”7 

•  C-5  Reliability  Enhancement  and  Reengineering  Program  (RERP): 
“The  Air  Force’s  C-5  RERP  is  one  of  two  major  upgrades  for  the  C- 
5.  The  RERP  is  designed  to  enhance  the  reliability  of  the  aircraft 
through  the  replacement  of  engines  and  modifications  to 
subsystems  such  as  the  electrical,  fuel,  hydraulic  and  flight  controls 
systems,  while  the  C-5  Avionics  Modernization  Program  (AMP)  is 
designed  to  enhance  the  avionics.  These  upgrades  are  part  of  a 
two-phased  modernization  effort  to  improve  the  mission  capability 
rate,  transport  capabilities  and  reduce  ownership  costs.”8 

•  F/A-22  Raptor:  “The  Air  Force’s  F/A-22,  originally  planned  to  be  an 
air  superiority  fighter,  will  also  have  air-to-ground  attack  capability. 
It  is  being  designed  with  advanced  features,  such  as  stealth 

5  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  31. 

6  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  33. 

7  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  35. 

8  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  37. 
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characteristics,  to  make  it  less  detectable  to  adversaries  and 
capable  of  high  speeds  for  long  ranges.  It  also  has  integrated 
aviation  electronics  (avionics)  designed  to  greatly  improve  pilots’ 
awareness  of  the  situation  surrounding  them.  It  is  designed  to 
replace  the  Air  Force’s  F-15  aircraft.”9 

•  Global  Hawk:  “The  Air  Force’s  Global  Hawk  system  is  a  high 
altitude,  long  endurance  unmanned  aerial  vehicle  with  integrated 
sensors  and  ground  stations  providing  intelligence,  surveillance, 
and  reconnaissance  capabilities.”10 

•  Joint  Strike  Fighter  (JSF):  “The  program  goals  are  to  develop  and 
field  a  family  of  stealthy,  strike  fighter  aircraft  for  the  Navy,  Air 
Force,  Marine  Corps,  and  U.S.  allies,  with  maximum  commonality 
to  minimize  costs.  The  carrier  suitable  version  will  complement  the 
Navy’s  F/A-18  E/F.  The  conventional  take-off  and  landing  version 
will  primarily  be  an  air-to-ground  replacement  for  the  Air  Force’s  F- 
16  and  A-10  aircraft,  and  will  complement  the  F/A-22.  The  short 
take-off  and  vertical  landing  version  will  replace  the  Marine  Corps’ 
F/A-18  and  AV-8B  aircraft.”11 

•  Joint  Unmanned  Combat  Air  Systems  (JUCAS):  “The  program  is  a 
combined  effort  of  the  Defense  Advanced  Research  Projects 
Agency  (DARPA),  the  Air  Force,  and  the  Navy  to  demonstrate  the 
technical  feasibility  and  operational  value  of  a  networked  system  of 
high  performance  and  weaponized  unmanned  air  vehicles. 
Expected  missions  include  the  suppression  of  enemy  air  defenses, 
electronic  attack,  precision  strike,  and  surveillance.  The  program 
consolidates  two  formerly  separate  service  projects  and  is  to 
develop  larger,  more  capable,  and  interoperable  aircraft.”12 

•  Predator:  “The  Air  Force’s  MQ-9  Predator  B  is  a  multi-role, 
medium-to-high  altitude  endurance  unmanned  aerial  vehicle 
system  capable  of  flying  at  higher  speeds  and  higher  altitudes  than 
its  predecessor  the  MQ-1  Predator  A.  The  Predator  B  is  designed 
to  provide  a  ground  attack  capability  and  will  employ  fused  multi- 
spectral  sensors  to  find  and  track  small  ground  mobile  or  fixed 
targets.  As  envisioned,  each  Predator  B  system  will  consist  of  four 
aircraft,  a  ground  control  station,  and  a  satellite  communication 
suite.”13 


9  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  63. 

10  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  67. 

1 1  GAO-05-301 ,  Assessments  of  Selected  Major  Weapon  Programs,  p.  79. 

12  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  87. 

13  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  101 
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•  V- 22  Joint  Services  Advanced  Vertical  Lift  Aircraft:  “The  V-22 

Osprey  is  a  tilt  rotor,  vertical  takeoff  and  landing  aircraft  being 
developed  by  the  Navy  for  Joint  Service  application.  It  is  designed 
to  meet  the  amphibious/vertical  assault  needs  of  the  Marine  Corps, 
the  strike  rescue  needs  of  the  Navy,  and  the  special  operations 
needs  of  the  Air  Force  and  the  U.S.  Special  Operations  Command. 
The  MV-22  version  will  replace  the  CH-46E  and  CH-53D 
helicopters  of  the  Marine  Corps.”14 

4.  The  Technology  Development  Phase 

The  figure  below15  show  the  phases  of  the  weapon  system  development 
and  acquisition  process,  as  described  in  DoDI  500.2.  The  Technology 
Development  Phase  is  a  specific  phase  of  this  process,  and  the  following 
descriptions,16  in  Table  1,  provide  a  more  detailed  description  of  the  Technology 
Development  Phase. 


User  Needs  & 
Technology  Opportunities 


✓  4 

A  B 
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Production  & 
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C  )  Readiness 

V  Review 

/\  FRP 
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IOC  FOC 


Figure  1.  The  Technology  Development  Phase(From:  DoD  5000.2,  Defense 

Acquisition  Guidebook,  -  3.  Procedures) 


14  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  117. 

15  DoD  5000.2,  Defense  Acquisition  Guidebook,  -  3.  Procedures,  Figure  1,  Framework. 

16  DoD  5000.2,  Defense  Acquisition  Guidebook,  -  3.6.  Technology  Development. 
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a.  Purpose 

The  purpose  of  the  technology-development  phase  is  to  reduce  technology  risk  and  to  determine 
the  appropriate  set  of  technologies  to  be  integrated  into  a  full  system.  Technology  Development 
is  a  continuous  technology  discovery  and  development  process  reflecting  close  collaboration 
between  the  S&T  community,  the  user,  and  the  system  developer.  It  is  an  iterative  process 
designed  to  assess  the  viability  of  technologies  while  simultaneously  refining  user  requirements. 

b.  Milestone  A 

The  project  shall  enter  technology  development  at  Milestone  A  when  the  MDA  has  approved  the 
Technology  Development  Strategy  (TDS).  The  tables  in  Enclosure  3  identify  all  statutory  and 
regulatory  requirements  applicable  to  Milestone  A.  This  effort  normally  shall  be  funded  only  for 
the  advanced  development  work.  A  favorable  Milestone  A  decision  DOES  NOT  mean  that  a  new 
acquisition  program  has  been  initiated. 

c.  Shipbuilding  programs 

Shipbuilding  programs  may  be  initiated  at  the  beginning  of  Technology  Development.  The 
information  required  in  the  tables  at  Enclosure  3  shall  support  program  initiation.  A  cost 
assessment  shall  be  prepared  in  lieu  of  an  independent  cost  estimate  (ICE),  and  a  preliminary 
assessment  of  the  maturity  of  key  technologies  shall  be  provided. 

d.  Technology  consideration 

The  Interim  Capability  Document  (ICD)  and  the  Technology  Development  Strategy  (TDS)  shall 
guide  this  effort.  Multiple  technology-development  demonstrations  may  be  necessary  before  the 
user  and  developer  agree  that  a  proposed  technology  solution  is  affordable,  militarily  useful,  and 
based  on  mature  technology.  The  TDS  shall  be  reviewed  and  updated  upon  completion  of  each 
technology  spiral  and  development  increment.  Updates  shall  be  approved  to  support  follow-on 
increments. 

e.  Evolutionary  acquisition 

If  an  evolutionary  strategy  is  used,  the  initial  capability  represents  only  partial  fulfillment  of  the 
overall  capability  described  in  the  Interim  Capability  Document  (ICD),  and  successive  technology 
development  efforts  continue  until  all  capabilities  have  been  satisfied.  In  an  evolutionary 
acquisition,  the  identification  and  development  of  the  technologies  necessary  for  follow-on 
increments  continues  in  parallel  with  the  acquisition  of  preceding  increments,  allowing  the  mature 
technologies  to  more  rapidly  proceed  into  System  Development  and  Demonstration  (SDD).  Each 
increment  of  an  evolutionary  acquisition  program  shall  have  an  associated  MDA-approved  TDS. 

f.  Exiting  the  phase 

The  project  shall  exit  technology  development  when  an  affordable  increment  of  militarily  useful 
capability  has  been  identified,  the  technology  for  that  increment  has  been  demonstrated  in  a 
relevant  environment,  and  a  system  can  be  developed  for  production  within  a  short  timeframe 
(normally  less  than  five  years);  or  when  the  MDA  decides  to  terminate  the  effort.  During 
Technology  Development,  the  user  shall  prepare  the  Capability  Development  Document  (CDD)  to 
support  program  initiation,  refine  the  integrated  architecture,  and  clarify  how  the  program  will  lead 
to  joint  war  fighting  capability.  The  CDD  builds  on  the  ICD  and  provides  the  detailed  operational 
performance  parameters  necessary  to  design  the  proposed  system.  A  Milestone  B  decision 
follows  the  completion  of  Technology  Development. 

Table  1.  Detailed  Description  of  the  Technology  Development  Phase  (From: 
DoD  5000.2,  Defense  Acquisition  Guidebook,  -  3.6.  Technology 

Development) 
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B.  COST  ESTIMATING17 

Cost  analysis  is:  (1)  the  act  of  developing,  analyzing,  and 
documenting  cost  estimates  using  analytical  approaches  and 
techniques.  (2)  The  process  of  analyzing  and  estimating 
incremental  and  total  resources  required  to  support  past,  present, 
and  future  forces,  units,  systems,  functions,  and  equipment.  It  is 
an  integral  step  in  the  selection  between  alternatives  by  the 
decision  maker.  (3)  A  management  tool  used  to  help  decision 
makers  evaluate  resource  requirements  at  key  management 
milestones  and  decision  points  in  the  acquisition  process.  Cost 
analysis  is  used  to  produce  cost  estimates  for  materiel  systems, 
automated  information  systems,  force  units,  training,  and  other 
Army  programs  and  projects.  Each  cost  analysis  should  contain: 

(1 )  a  clear  definition  of  what  is  being  costed. 

(2)  The  specification  of  all  assumptions,  ground  rules,  and 
constraints,  assumed  or  imposed,  underlying  the  analysis. 
They  must  each  be  explained  with  adequate  rationale. 

(3)  An  estimate  of  all  expected  costs,  directly  or  indirectly 
associated  with  the  project  over  its  life,  including  disposal. 

The  cost  estimate  must  include  the  identification  of  all  data 
sources  used. 

(4)  Risk  and  uncertainty  analyses  identifying  any  circumstances 
which  could  affect  a  course  of  action. 

(5)  Key  limitations  in  terms  of  elements  that  were  excluded. 

The  documentation  supporting  the  cost  analysis  should  describe 
the  methodology  used  in  developing  these  estimates.  It  also 
should  identify  all  the  data  sources  and  include  the  computations 
used  to  estimate  the  costs.  The  documentation  should  be  in 
sufficient  detail  to  permit  reviewers  to  follow  the  logic  from 
assumptions  to  conclusion  and  to  update  the  estimate  at  a  later 
time. 

Cost  analysis  is  a  critical  element  in  the  DoD  acquisition  process. 

It  supports  management  decisions  by  quantifying  the  resource 
impact  of  alternative  options.  A  quality  analysis  includes  different 
acquisition  strategies,  hardware  designs,  software  designs, 
personnel  requirements,  and  operating  and  support  concepts.  As 
a  program  matures  and  more  information  becomes  available,  the 
cost  estimate  grows  in  complexity  and  detail.  One  test  of  the  utility 

17  Department  of  the  Army  Cost  Analysis  Manual,  U.S.  Army  Cost  and  Economic  Analysis 
Center,  May  2002. 
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of  cost  analysis  is  its  ability  to  respond  quickly  to  program 
turbulence.  Army  planners  must  have  reliable  and  readily 
available  information  about  the  cost  consequences  of  program 
changes,  extensions,  or  cancellations.  Cost  analysts  must 
develop  models  to  support  these  quick  turnaround  analyses. 

Cost  analysis  has  an  on-going  role  in  the  management  of  base 
operations  as  well  as  acquisitions.  Cost  analysis  assists 
installations,  headquarters  in  determining  base  support 
requirements,  developing  budgets,  conducting  cost  benefit 
analysis,  and  performing  special  studies.  For  example,  the  office  of 
the  Deputy  Assistant  Security  of  Army  Cost  and  Economics  (DASA- 
CE)  develops  cost  factors  in  support  of  the  Army  Chief  of  Staff  for 
Installation  Management  (ACSIM)  for  both  the  Installation  Status 
Report  (ISR)  and  the  Army  Installation  Management  -  Headquarters 
Information  (AIM-HI)  model.  Other  ACSIM  efforts  supported  by 
cost  analysis  include  A-76  studies,  Service  Based  Costing,  and 
Standard  Service  Costing. 

With  the  establishment  of  the  cost/outcome  oriented  Government 
Performance  Results  Act  (GPRA),  cost  analysis  has  taken  on  a 
larger  role  in  to  support  management  of  base  operations.  The 
managerial  costing  focus,  to  meet  GPRA  mandates,  requires  cost 
analysis  in  the  measuring  and  management  of  cost  and  results. 
Cost  analysis  will  be  needed  to  develop  methodologies,  conduct 
studies  and  analyze  data  of  the  products  and  services  provided 
through  base  operations.  The  prerequisite  to  cost  management  is 
cost  measurement.  There  are  numerous  methods  of  measuring 
costs,  all  of  which  will  require  cost  analysis  skills  now  and  in  the 
future.  Examples  of  cost  measurement  include,  full  cost,  job-order 
cost,  service  based  cost,  activity  based  cost,  standard  cost,  product 
cost,  and  responsibility  cost  to  name  a  few.  Though  there  are 
many  examples  of  cost  measurement  each  demands  cost  analysis 
support  to  make  information  meaningful  to  DoD  management. 
DASA-CE  will  prepare  a  managerial  costing  manual  in  the  future  on 
Activity  Based  Costing,  Service  Based  Costing  and  Standard 
Service  Costing. 

Other  uses  of  cost  analysis  in  the  DoD  are  to: 

(1)  Support  decisions  on  program  viability,  structure,  and 
resource  requirements. 

(2)  Evaluate  the  cost  implications  of  alternative  materiel  system 
designs. 
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(3)  Provide  credible  and  auditable  cost  estimates  in  support  of 
milestone  reviews  during  the  acquisition  process. 

(4)  Assess  the  cost  implications  of  new  technology,  new 
equipment,  new  force  structures,  or  new  operating  or 
maintenance  concepts. 

(5)  Support  the  Planning,  Programming,  Budgeting,  and 
Execution  System  (PPBES)  process.  This  includes 
formulating  and  documenting  Army  Cost  Positions  (ACPs) 
on  programs  within  the  Program  Objective  Memorandum 
(POM)  and  the  Budget  Estimate  Submission  (BES) 
processes. 

(6)  Determine  the  funds  required  for  a  given  level  of  training  or 
operational  activity  such  as  miles  driven  per  year. 

Cost  analysis  applies  scientific  and  statistical  methods  to  evaluate 
the  likely  cost  of  a  specific  item  in  a  defined  scenario.  In  the  real 
world,  there  are  multiple  uncertainties  about  the  item's  cost.  Some 
"internal"  uncertainties  influencing  cost  are  inadequate  item 
definition,  poor  contract  statement  of  work,  optimistic  proposed 
solutions,  inexperienced  management,  and  success-oriented 
scheduling.  Some  "external"  uncertainties  include  funding 
turbulence,  contractor's  underestimating  of  complexity,  contractor's 
changing  business  base,  and  excessive  (or  insufficient) 
Government  oversight.  In  spite  of  uncertainty,  the  process  of  cost 
analysis  is  the  most  rigorous  approach  available  to  evaluate  the 
costs  of  alternatives  for  the  decision  maker.  Cost  analysis  does 
have  limitations.  Analysts  develop  cost  estimating  methodologies 
with  an  imperfect  understanding  of  the  technical  merits  and 
limitations  of  the  item.  The  applicability  of  historic  data  is  always 
subject  to  interpretation.  Because  of  future  uncertainties,  there 
are  limitations  in  determining  the  degree  to  which  reality  varies  from 
the  plan.  Realistically,  the  cost  analysis  process  cannot: 

(1)  Be  applied  with  cookbook  precision,  but  must  be  tailored  to 
the  problem. 

(2)  Produce  results  that  are  better  than  input  data. 

(3)  Predict  political  impacts. 

(4)  Substitute  for  sound  judgment,  management,  or  control. 

(5)  Make  the  final  decisions. 
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Despite  these  limitations,  cost  analysis  is  a  powerful  tool. 
Rigorous  and  systematic  analysis  leads  to  a  better  understanding 
of  the  problem.  It  improves  management  insight  into  resources 
allocation  problems.  Because  the  future  is  uncertain  our  best 
estimate  will  differ  from  reality. 

C.  GAO  METHODOLOGY 

Our  review  of  the  “best  practices”  literature  in  weapons-acquisition 
programs  provides  the  lesson  that  three  considerations  are  critical  for  sound 
progress  in  the  weapons-acquisition  process.  These  are 

•  Technology  maturity 

•  Design  stability 

•  Controlled  production  stability 

Technology  development  is  the  key  factor  in  determining  whether  a 
weapons-development  program  could  meet  requirements  without  cost  overruns 
and  schedule.  Acquisition-program  management  also  relies  on  the  knowledge- 
based  approach  in  developing  new  weapon  programs  and  ensures  key  results 
through  high  levels  of  knowledge. 

The  US  GAO  methodology  implements  three  knowledge  points,  as 
discussed  below,  and  the  GAO  writes  that 

The  attainment  of  each  successive  knowledge  point  builds  on  the 
preceding  one.  While  the  knowledge  itself  builds  continuously 
without  clear  lines  of  demarcation,  the  attainment  of  knowledge 
points  is  sequential.  In  other  words,  production  maturity  cannot  be 
attained  if  the  design  is  not  stable,  and  design  stability  cannot  be 
attained  if  the  critical  technologies  are  not  mature.18 

These  knowledge  points  are  described  in  the  paragraphs  below. 

1.  Knowledge  Point  One:  Technology 

Technology  Readiness  Levels  (TRLs),  used  in  the  US  National 
Aeronautics  and  Space  Administration’s  (NASA)  planning  for  many  years,  are 
measured  on  a  scale  of  nine  levels  of  technological  maturity,  as  shown  in  Table 
2. 19  Knowledge  point  one  occurs  when  the  weapon-acquisition  requirements 

18  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  7. 

19  Technology  Readiness  Level  Definitions, 
http://www.globalsecuritv.org/militarv/intro/trl.htm  (accessed  November  29,  2005) 
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meet  the  developer’s  available  resources  in  terms  of  knowledge,  time,  and 
budget.  The  availability  of  mature  technology  on  which  to  base  development 
provides  enhanced  probability  of  achieving  project  success  from  the  beginning. 
TRLs  have  been  adapted  to  facilitate  comparison  between  TRLs  at  the 
acquisition  level  and  TRLs  at  the  parts  level.  To  identify  critical  levels  of 
technology,  GAO  attempts  to  find  the  key  technologies  in  which  information 
availability  may  be  a  concern.  As  shown  in  the  GAO  report,  “our  best  practices 
work  has  shown  that  a  technology  readiness  level  of  7 — demonstration  of  a 
technology  in  an  operational  environment — is  the  level  of  technology  maturity 
that  constitutes  a  low  risk  of  starting  a  product  development  program.”20 


Technology 
Readiness  Level 

Description 

1.  Basic  principles 
observed  and 
reported 

Lowest  level  of  technology  readiness.  Scientific  research  begins  with  to  be 
translated  into  applied  research  and  development.  Example  might  include  paper 
studies  of  a  technology's  basic  properties. 

2.  Technology  concept 
and/or  application 
formulated 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can 
be  invented.  The  application  is  speculative  and  there  is  no  proof  or  detailed 
analysis  to  support  the  assumption.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  and 
experimental  critical 
function  and/or 
characteristic 

Active  research  and  development  is  initiated.  This  includes  analytical  studies 
and  laboratory  studies  to  physically  validate  analytical  predictions  of  separate 
elements  of  the  technology.  Examples  include  components  that  are  not  yet 
integrated  or  representative. 

4.  Component  and/or 
breadboard 
validation  in 
laboratory 
environment 

Basic  technological  components  are  integrated  to  establish  that  the  pieces  will 
work  together.  This  is  relatively  "low  fidelity"  compared  to  the  eventual  system. 
Examples  include  integration  of 'ad  hoc'  hardware  in  a  laboratory. 

5.  Component  and/or 
breadboard 
validation  in  relevant 
environment 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic  supporting 
elements  so  that  the  technology  can  be  tested  in  a  simulated  environment. 
Examples  include  'high  fidelity'  laboratory  integration  of  components. 

6.  System/subsystem 
model  or  prototype 
demonstration  in  a 
relevant 
environment 

Representative  model  or  prototype  system,  which  is  well  beyond  the 
breadboard  tested  for  TRL  5,  is  tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology's  demonstrated  readiness.  Examples  include 
testing  a  prototype  in  a  high  fidelity  laboratory  environment  or  in  simulated 
operational  environment. 

20  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  130. 
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Technology 
Readiness  Level 

Description 

7.  System  prototype 
demonstration  in  a 
operational 
environment 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up 
from  TRL  6,  requiring  the  demonstration  of  an  actual  system  prototype  in  an 
operational  environment,  such  as  in  an  aircraft,  vehicle  or  space.  Examples 
include  testing  the  prototype  in  a  test  bed  aircraft. 

8.  Actual  system 
completed  and  'flight 
qualified'  through 
test  and 
demonstration 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected 
conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of  true  system 
development.  Examples  include  developmental  test  and  evaluation  of  the 
system  in  its  intended  weapon  system  to  determine  if  it  meets  design 
specifications. 

9.  Actual  system  'flight 
proven'  through 
successful  mission 
operations 

Actual  application  of  the  technology  in  its  final  form  and  under  mission 
conditions,  such  as  those  encountered  in  operational  test  and  evaluation.  In 
almost  all  cases,  this  is  the  end  of  the  last  "bug  fixing"  aspects  of  true  system 
development.  Examples  include  using  the  system  under  operational  mission 
conditions. 

Table  2.  Definition  of  Technology  Readiness  Level  (From: 

http://www.qlobalsecuritv.orq/militarv/intro/trl.htm,  accessed  November  29, 

2005) 

2.  Knowledge  Point  Two:  Design 

This  knowledge  point  occurs  at  the  point  when  a  product's  design 
drawings  are  released  from  the  engineer  to  the  manufacturer.  Successful 
programs  obtain  design  stability  at  the  time  of  critical  design  review  (CDR),  and 
CDR  results  serve  as  criteria  for  estimating  the  stability  of  engineering  drawings 
released.  The  benchmark  that  the  US  GAO  has  set  is  that  engineering-drawing 
stability  is  achieved  if  it  is  true  that  at  least  90%  of  engineering  drawings  are 
completed  (or  at  least  releasable)  before  the  CDR. 

3.  Knowledge  Point  Three:  Production 

Products  manufactured  with  statistical  process  controls  in  place  to  govern 
costs,  and  schedules  represent  maturity  in  the  production  process.  To  evaluate 
such  maturity,  critical  manufacturing  processes  are  monitored  for  statistical  data 
and  fractions  of  statistical  process  control.  To  ascertain  status  of  production,  two 
concepts  should  be  considered:  quality  and  product  testing. 

•  Quality  in  the  manufacturing  processes  requires  a  repeatable 
statistical  approach  to  ensure  that  standards  for  the  system  have 
been  defined  and  are  being  monitored.  Production  must  be  on  time, 
meet  schedules,  and  keep  in  cost. 
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•  Product  test  is  needed  to  guarantee  that  the  system  operates 
reliably,  without  failure  or  repair  during  the  some  period.  Production 
tests  require  many  trials  and  processes. 

Due  to  the  typical  shortage  and  high  cost  of  the  personnel  needed  to 

perform  such  tests,  we  focus  only  on  the  critical  processes  necessary  to  meet  the 

reliability  and  performance  standards.  To  measure  the  performance  of  the 

process,  a  process-capability  index  (CPK)  is  used  to  determine  how  a  process  is 

running  compared  to  its  specification  limits  and  to  measure  production-process 

capability  within  specified  limits. 

GAO  used  the  Process  Capability  Index  which  is  a  process 
performance  measurement  that  quantifies  how  closely  a  process  is 
running  to  its  specification  limits.  The  index  can  be  translated  into 
an  expected  product  defect  rate,  and  GAO  has  found  it  to  be  a  best 
practice.21 

The  CPK  and  probability  of  a  defective  part  are  shown  in  the  table 
below.22  A  standard  of  less  than  1.33  indicates  that  the  manufacturing  process 
lacks  statistical  control  and  acceptable  consistency.  Satisfactory  cost,  schedule, 
and  product  quality  could  be  obtained  during  the  manufacturing  process  prior  to 
production. 


Manufacturing  process  capability  index 

Associated  defect  rate 

Cpk  -  .67  (not  capable) 

1  in  22  parts  produced 

Cpk  -  1 .0  (marginally  capable) 

1  in  370  parts  produced 

Cpk  - 1 .33  (industry  standard) 

1  in  15,152  parts  produced 

Cpk  -  2.0  (industry  growth  goal) 

1  in  500,000,000  parts  produced 

Table  3.  CPK  and  Probability  of  a  Defective  Part  (From:  GAO  02-701 ) 


21  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  131. 

22  GAO  02-701,  Best  Practices,  p.  39. 
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a.  Process  Capability  Index23 

Process  capability  compares  the  output  of  an  in-control  process  to 
the  specification  limits  by  using  capability  indices.  The  comparison  is  made  by 
forming  the  ratio  of  the  spread  between  the  process  specifications  (the 
specification  "width")  to  the  spread  of  the  process  values,  as  measured  by  six 
process  standard-deviation  units  (the  process  "width").  A  process  capability  index 
uses  both  the  process  variability  and  the  process  specifications  to  determine 
whether  the  process  is  "capable."  A  capable  process  is  one  where  almost  all  the 
measurements  fall  inside  the  specification  limits.  This  can  be  represented 
pictorially  by  the  plot  below. 


aHeuia&le  process  spread 


Figure  2.  The  Specification  Limits  (From: 
http://www.itl.nist.gov/div898/handbook/pmc/section1/pmc16.htm 
accessed  November  29,  2005) 

There  are  several  statistics  that  can  be  used  to  measure  the 
capability  of  a  process:  Cp,  Cpk,  Cpm. 

Most  capability  indices  estimates  are  valid  only  if  the  sample  size 
used  is  large  enough,  generally  thought  to  be  about  50  independent  data  values. 


23  Engineering  Statistics  Book, 

http://www.itl.nist.gov/div898/handbook/pmc/section1/pmc16.htm  (accessed  November  29,  2005). 
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The  Cp,  Cpk,  and  Cpm  statistics  assume  that  the  population  of  data 
values  is  normally  distributed.  Assuming  a  two-sided  specification,  if  ft  and  cj 
are  the  mean  and  standard  deviation,  respectively,  of  the  normal  data  and  USL, 
LSL,  and  T  are  the  upper  and  lower  specification  limits  and  the  target  value, 
respectively,  then  the  population  capability  indices  are  defined  as  follows: 


USL  -  LSL 


^  fi-Lh'L , 

Cpk  =  mm  [ - — - , - — - J 


Cpm  — 


3(t  ?  3<r 

USL -LSL 

^y/a2  +  (fi-T)2 


Sample  estimators  for  these  indices  are  given  below.  (Estimators 
are  indicated  with  a  "hat"  over  them). 


0p  = 


USL  -  LSL 
6s 


.  rUSL  —  x  x-LSL , 
=  mm  [ - — - , - - J 


3* 


3s 


USL -LSL 
6^*4- (5 -T)8 


The  estimator  for  Cpk  can  also  be  expressed  as  Cpk  =  Cp(1-k), 
where  k  is  a  scaled  distance  between  the  midpoint  of  the  specification  range,  m, 
and  the  process  mean,  /a. 


Denote  the  midpoint  of  the  specification  range  by  m  = 
(USL+LSL)/2.  The  distance  between  the  process  mean,  ft,  and  the  optimum, 
which  is  m,  is  /a-  m,  where  rn<  /a  <  LSL.  The  scaled  distance  is 

fc-  lm~^l  0<*-<1 
(i vsL-LSL){r  ~ 

(the  absolute  sign  takes  care  of  the  case  when  LSL  <  /j  <  rre).  To  determine 
the  estimated  value,  fc,  we  estimate  ft  by  S.  Note  that  x  <  USL. 
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The  estimator  for  the  Cp  index,  adjusted  by  the  k  factor,  is 

Since  0  <  fc  <  1,  it  follows  that  Cfk  <  c„ 

To  get  an  idea  of  the  value  of  the  Cp  statistic  for  varying  process 
widths,  consider  the  following  plot: 


USL - LSL 

6o 

8a 

10a 

12a 

cp 

1.00 

1.33 

1.66 

2.00 

Rejects 

.21% 

64  ppm 

.6  ppm 

2  ppb 

%  of  spec  used 

100 

75 

60 

50 

Figure  3.  Varying  Process  Widths  (From: 
http://www.itl.nist.gov/div898/handbook/pmc/section1/pmc16.htm 
accessed  November  29,  2005)) 
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This  can  be  expressed  numerically  by  the  table  below  where  ppm  = 
parts  per  million  and  ppb  =  parts  per  billion.  Note  that  the  reject  figures  are  based 
on  the  assumption  that  the  distribution  is  centered  at  p. 


We  have  discussed  the  situation  with  two  spec  limits:  the  USL  and 
LSL.  This  is  known  as  the  bilateral  or  two-sided  case.  There  are  many  cases 
where  only  the  lower  or  upper  specifications  are  used.  Using  one  spec  limit  is 
called  unilateral  or  one-sided.  The  corresponding  capability  indices  are 


g  allowable  upper  spread  USL 

**  actual  upper  spread 


3ct 


and 


allowable  lower  spread 
actual  lower  spread 


fi-LSL 


where  /i  and  a  are  the  process  mean  and  standard  deviation,  respectively.  We 

A 

would  like  to  have  cp*  at  least  1.0.  Below  1.0  means  this  is  not  a  good 
process.  If  possible,  reduce  the  variability  or/and  center  the  process.  We  can 

n  f* 

compute  the  C?!i  and  Estimators  of  Cpu  and  Cpi  are  obtained  by  replacing 
and  o  by  x  and  s,  respectively.  The  following  relationship  holds  Cp  =  (Cpu  + 
Cpi)  /2. 


This  can  be  represented  pictorially  by: 
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artuai  WUri 

t _ i 

aBotu  able  upper  spregd 

Figure  4.  Actual  and  Allowable  Upper  Spread  (From: 
http://www.itl.nist.gov/div898/handbook/pmc/section1/pmc16.htm 
accessed  November  29,  2005)) 

Note  that  we  also  can  write:  Cpk  =  min  {Cpi,  Cpu}.” 


b.  Manufacturing  and  Product  Reliability  Knowledge  of 
GAO  Report 

The  decision  maker  must  decide  whether  the  weapons-acquisition 
program  is  ready  to  make  the  transition  from  development  into  production.  The 
below  Table  424  would  provide  the  guidance  necessary  to  making  a  successful, 
data-driven,  decision  process  in  the  weapon-acquisition  program,  thereby 


reducing  the  possibility  of  delaying  the  schedule  and  increasing  costs. 


Identify  key  system 
characteristics  and 
critical  manufacturing 
processes. 

Key  product  characteristics  and  critical  manufacturing 
processes  are  identified.  Because  there  can  be  thousands  of 
manufacturing  processes  required  to  build  a  product, 
companies  focus  on  the  critical  processes — those  that  build 
parts  that  influence  the  product’s  key  characteristics  such  as 
performance,  service  life,  or  manufacturability. 

Determine  processes  in 
control  and  capability. 

Statistical  process  control  is  used  to  determine  if  the 
processes  are  consistently  producing  parts.  Once  control  is 
established,  an  assessment  is  made  to  measure  the  process’s 
ability  to  build  a  part  within  specification  limits  as  well  as  how 
close  the  part  is  to  that  specification.  A  process  is  considered 
capable  when  it  has  a  defect  rate  of  less  than  one  out  of  every 

24  GAO  02-701,  Best  Practices,  p.  38. 


15,152  parts  produced. 

Conduct  failure  modes 
and  effects  analysis. 

Bottom-up  analysis  is  done  to  identify  potential  failures  for 
product  reliability.  It  begins  at  the  lowest  level  of  the  product 
design  and  continues  to  each  higher  tier  of  the  product  until 
the  entire  product  has  been  analyzed.  It  allows  early  design 
changes  to  correct  potential  problems  before  fabricating 
hardware. 

Set  reliability  growth 
plan  and  goals. 

A  product’s  reliability  is  its  ability  to  perform  over  an  expected 
period  of  time  without  failure,  degradation,  or  need  of  repair.  A 
growth  plan  is  developed  to  mature  the  product’s  reliability 
over  time  through  reliability  growth  testing  so  that  it  has  been 
demonstrated  by  the  time  production  begins. 

Conduct  reliability 
growth  testing. 

Reliability  growth  is  the  result  of  an  iterative  design,  build,  test, 
analyze,  and  fix  process  for  a  product’s  design  with  the  aim  of 
improving  the  product’s  reliability  overtime.  Design  flaws  are 
uncovered  and  the  design  of  the  product  is  matured. 

Conduct  executive  level 
review  to  begin 
production. 

Corporate  stakeholders  meet  and  review  relevant  product 
knowledge,  including  manufacturing  and  reliability  knowledge, 
to  determine  whether  a  product  is  ready  to  begin  production. 

The  decision  is  tied  to  the  capture  of  knowledge. 

Table  4.  Activities  to  Capture  Manufacturing  Knowledge  and  Make 

Decisions  (After:  GAO  02-701) 

D.  EVALUATING  AFFORDABILITY  INITIATIVES 

There  are  different  approaches  to  determining  technology  maturity.  For 
example,  there  are  those  developed  by  the  Joint  Strike  Fighter  (JSF)  Program 
Office’s  (PO)  Advanced  Cost  Estimating  (ACE)  Integrated  Product  Team  (IPT)  to 
clarify  their  technology  initiatives.  The  description  below25  shows  the  status  and 
definition  of  technology  maturity  factors,  as  well  as  the  sub-factors26  that  support 
them. 


Definitions  for  maturity  factors  were  considered  carefully  by  IPT 
members.  All  scales  range  from  very  easy  (a  known,  presently 
operational  technology,  design,  or  process)  to  extremely  difficult 
(purely  theoretical  concept  or  laboratory  research  yet  to  be 
attempted  in  a  production  environment). 

The  state-of-technology  sub-factor  measures  each  initiative  in 
terms  of  availability  and  promise  of  the  technology  required  for 

25  Johns  Hopkins  APL  Technical  Digest,  volume21,  number  3  (2000),  Evaluating 
Affordability  Initiatives,  pp.  429-430. 

26  Johns  Hopkins  APL  Technical  Digest,  volume  21,  number  3  (2000),  Evaluating 
Affordability  Initiatives,  p.  430. 
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success.  Ability  to  attain  the  required  level  of  technological 
sophistication  within  known  schedule  constraints,  as  well  as 
hardware  and  testing  maturity,  is  considered.  Although 
assessments  were  performed  on  a  “system”  basis,  participants 
were  also  asked  to  consider  and  comment  on  the  technological 
requirements  of  key  subsystems.  The  scale  ranges  from  technology 
that  is  already  operational  and  deployed  to  that  which  still  requires 
significant  scientific  research 

The  design  and  engineering  sub-factor  measures  each  initiative  in 
terms  of  difficulty  in  advancing  the  state  of  the  art  to  that  required 
for  the  JSF  Program.  Thus,  design  and  engineering  focuses  on 
implementation,  separate  from  technology.  For  example,  an  item 
previously  engineered  and  production-qualified  may  require 
extensive  form,  fit,  and  function  changes  or  modifications  for  JSF 
application.  Key  subsystems  are  also  considered.  The  scale  ranges 
from  off-the-shelf  items  meeting  all  requirements  to  ones  requiring 
new,  breakthrough  design  or  engineering  efforts. 

The  manufacturing  process  sub-factor  measures  each  initiative  in 
terms  of  the  process  capability  needed  to  produce  required 
quantities  for  the  JSF  Program.  Process  attributes  considered  are 
metrics  such  as  number  of  allowable  defects,  yield  or  throughput 
requirements,  tolerance  or  precision  requirements,  overall  process 
capability  to  be  maintained,  and  allowable  failure  rates.  Evaluation 
includes  assessment  of  advances  required  to  move  from  current  to 
proposed  manufacturing  and  assembly  processes.  The  scale 
ranges  from  an  existing  demonstrated  process  that  satisfies  all  key 
attributes  to  one  that  exceeds  the  state  of  the  art  for  at  least  one 
key  attribute. 

The  resource  availability  sub-factor  grades  an  initiative  on  the 
availability  of  all  resource  elements  necessary  for  implementation  at 
production  quantities  and  rates  specified  for  the  JSF  Program. 
Resources  include  parts  or  subassemblies,  tools  and  fixtures,  test 
equipment  and  facilities,  personnel  (including  their  skills  and 
training  levels),  materials  (quantity  and  quality),  production 
equipment  and  facilities,  and  funding.  The  scale  ranges  from  all 
required  resources  being  readily  available  through  at  least  one 
dependable  source  to  an  initiative  for  which  resources  have  not  yet 
been  specified. 
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Level 

State  of 
technology 

Design  and 
engineering 

Manufacturing 

process 

Resource  availability a 

A 

Initiative 
presently 
operational  and 
deployed 

“Off-the  shelf 
hardware 
requirements 

Existing, 

demonstrated 

process 

Readily  obtainable  through  at 
least  one  source;  successful 
past  experience 

B 

Process  in 

limited 

operation 

Design 

required; 

existing 

components, 

specs 

Modified; 

within 

demonstrated 

norms 

All  but  one  resource 
obtainable  from  at  least  one 

source 

c 

Process  passed 
acceptance; 
approved  for 
limited 
operation 

Design 
required; 
beyond 
present  specs 

New 

combination  of 

demonstrated 

processes 

All  but  two  resources 
obtainable  from  at  least  one 

source 

D 

Process  passed 
qualification  tests 

Some 

development 
effort  required 

Demonstrated, 
but  one  key 
attribute  new 

More  than  two  resources 
obtainable  from  at  lest  one 

source 

E 

Process  passed 

performance 

tests/in 

qualification 

testing 

Moderate 
development 
effort  required 

Demonstrated, 
but  two  or  more 
new  attributes 

Only  one  resource  obtainable 
from  at  least  one  source 

F 

Process 

feasibility 

demonstrated 

Major 

development 
effort  required 

New  process; 
within 

state  of  the  art 

Resources  have  been 
specified 

G 

Prototype  system 
tested;  significant 
scientific 
research 
required 

"Breakthrough” 
development 
effort  required 

New  process; 
exceeds  state  of 
the  art 

Resources  have  not  been 
specified 

A  Resource  include  tools/futures,  test  equipment,  personnel,  materials,  and  facilities. 


Table  5.  Definitions  of  Maturity  Sub  Factors  (From:  Johns  Hopkins  APL 
Technical  Digest,  volume  21,  number  3  (2000),  Evaluating  Affordability 

Initiatives) 
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E.  SUMMARY 

This  chapter  highlighted  the  features  of  the  related  research.  The 
technology-development  phase  reflecting  the  technology  discovery  processes, 
addresses  technology  risk  and  stage  of  development  needed  to  satisfy  the  user 
requirements.  GAO  methodology  also  suggests  the  three  points  in  time  at  which 
to  judge  technology  maturity,  and  these  judgments  play  a  significant  role  in 
judging  the  technological  risk  in  the  weapons-acquisition  program. 

•  First,  estimate  the  technology  maturity  from  the  development  start 
to  design  review  and  production.  The  technology  readiness  levels 
originally  developed  by  the  National  Aeronautics  and  Space 
Administration  may  be  used  as  criteria. 

•  The  second  technology  part,  design  stability,  will  be  estimated  by 
the  releasable  design  drawings  at  the  critical  design  review. 
Releasable  design  drawings  mean  the  completed  engineering 
drawings,  which  can  be  released  to  the  manufacturing  stage. 

•  The  third,  and  last,  technology  part,  production  maturity,  means  that 
critical  manufacturing  processes  are  under  control  of  statistical 
process  and  the  Process  Capability  Index  is  used  to  quantify  the 
statistical  process  of  the  product. 

The  manufacturing  and  product  reliability  knowledge  of  the  GAO  report 
shows  the  ideal  decision  process  to  meet  the  schedule,  budget  limitation,  and 
user’s  requirements  concerning  all  weapon  acquisition  program.  Yet  there  are 
different  approaches  to  determining  the  level  of  technology  maturity  by  examining 
the  affordability  of  weapons-system  development.  There  is  also  some  criticism 
about  GAO  methodology.  Even  though  GAO  methodology  could  be  used  with  the 
knowledge-based  approach,  Department  of  Defense  disagrees  with  GAO's 
March  05  assessment  of  JSF  technical  maturity  as  stated  in  GAO  reports  GAO- 
05-271  and  GAO-05-301.  Although  the  knowledge  points  provide  excellent 
indicators  of  potential  risks,  they  do  not  by  themselves  cover  all  elements  of  risk 
that  a  program  encounters  during  development,  such  as  funding  instability.  Our 
detailed  reviews  on  individual  systems  normally  provide  for  a  fuller  treatment  of 
risk  elements.27 


27  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  132. 
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III.  IMPLEMENTATION  AND  APPLICATION 


A.  TRL  ASSESSMENT 

1.  Technology  Assessment 

Tardy  arrival  of  key  technologies  can  significantly  delay  the  development 
of  a  weapon  system.  Testing  delays  and  cost  increases  in  the  F-22  program,  for 
example,  can  be  partly  traced  to  inadequate  maturity  of  Avionics,  as  measured 
by  TRLs.  To  prevent  a  similar  outcome  in  the  ROKAF  T-50  program,  program 
data  has  been  examined  and  GAO  knowledge  sets  applied,  and  the  resulting 
assessment  will  be  compared  against  the  US  program.  Because  judging  TRLs 
may  be  difficult  at  best,  gathering  up-to  date  information  about  the  program  as  a 
whole  is  critical.  We  requested  such  data  from  program  managers  and  made 
use  of  confidential  TRL  data  provided  to  NPS  about  the  U.S.  program.  Institute 
reports  on  the  T-50  will  rate  the  program  according  to  a  nine-level  technology- 
maturity  scale,  estimating  thirteen  critical  issues  and  649  technological  elements 
and  applying  GAO  standards,  by  which  “a  technology  readiness  level  of  7- 
demonstration  of  a  technology  in  an  operational  environment  -  is  the  level  of 
technology  maturity  that  constitutes  a  low  risk  of  starting  a  product  development 
program,”28  to  TRL  findings.  The  following  TRL  of  B-2RMP,  F-22  and  Predator 
were  derived  by  mapping  the  descriptions  of  technologies  from  GAO  reports. 
These  mappings  represent  the  author’s  own  views.  They  are  not  meant  to  be  an 
endorsement  of  any  official  views  and  others  may  view  the  matter  differently.  The 
others  are  the  reliable  data  from  program  manager  and  reports.  The  assessment 
of  T-50  TRLs  comes  from  “Analytic  evaluation  on  technology  acquisition  of  T-50 
program”  and  is  described  in  part  2  of  this  chapter. 


28  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  130. 
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Aircraft 

Development 

Design  review 

Production 

criteria 

B-2  RMP 

6.5 

9 

9 

7 

C-130  AMP 

7.7 

7.7 

8.5 

7 

C-5  AMP 

9 

9 

9 

7 

C-5  RERP 

8.1 

8.1 

8.1 

7 

F-22 

4 

4 

4 

7 

Global  Hawk 

5.8 

5.8 

5.8 

7 

JSF 

4.9 

4.9 

4.9 

7 

JUCAS 

3.7 

4.7 

4.7 

7 

Predator 

6.8 

6.8 

6.8 

7 

V- 22 

8 

8 

9 

7 

T-50 

6.3 

6.3 

6.3 

7 

Table  6.  TRL  Assessment 


B-2  C-130  C-5  C-5  F-22  Global  JSF  JUCA^redator  V-22  T-50 

RMP  AMP  AMP  RERP  Hawk 


Figure  5.  Development  TRL 
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B-2  C-130  C-5  C-5  F-22  Global  JSF  JUCA$>redator  V-22  T-50 

RMP  AMP  AMP  RERP  Hawk 


Figure  6.  Design  Review  TRL 
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B-2  C-130  C-5  C-5  F-22  Global  JSF  JUCA$>redator  V-22  T-50 

RMP  AMP  AMP  RERP  Hawk 


Figure  7.  Production  TRL 
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2.  Technology  Uncertainty 

Program  managers  of  weapon  acquisition  try  to  judge  the  level  program 
risk  by  using  TRLs.  Sometimes,  in  the  rapidly  growing  technology  environment,  a 
low  level  of  technology  in  the  beginning  of  the  development  phase  will  be 
sufficient  to  achieve  the  desired  TRL  in  the  design  review  and  production  phases. 
The  difficult  aspect  of  using  this  methodology  in  this  environment  is  the  challenge 
of  estimation  future  TRL.  “The  higher  the  level  of  technological  advancement,  the 
higher  the  level  of  opportunity  costs  for  a  successful  decision.  Greater 
technological  complexity  implies  a  greater  degree  of  uncertainty  or  ambiguity  for 
the  key  decision  maker”29  In  this  respect,  the  US  programs  also  have  to 
accommodate  uncertainty  in  forecasting  reasonable,  future  values  of  TRLs. 

The  data  given  by  US  program  managers  also  shows  the  difficulty  of 
estimating  the  TRL  in  the  design  review  and  production  period.  The  same 
challenge  is  true  in  the  case  of  T-50  program.  There  is  some  literature  which 
provides  a  method  for  estimation  future  values  of  TRLs.  For  forecasting  the 
maturation  of  technology  through  TRLs,  we  reviewed  the  report  by  Kirby  and 
Mavris.  Unfortunately,  this  paper  applies  only  to  TRL  1~5,  which  is  below  the 
range  of  interest  in  this  thesis.  Still,  the  conclusions  of  this  report  are  of  interest. 

This  paper  described  research  in  the  area  of  probabilistic 
technology  assessments  and  techniques  to  forecast  the  impact  of 
any  emerging  technology  in  the  conceptual  and  preliminary  phases 
of  aircraft  design.  The  thrusts  of  the  techniques  developed  were 
focused  on  the  description  of  technology  development  programs, 
and  the  various  milestones  encountered  during  a  successful 
program.  The  identification  of  sources  of  uncertainty  associated 
with  an  immature  technology  were  described  and  applied  to  the 
determination  of  frequency  distributions  of  a  technology’s  impact  on 
an  aerospace  system.30 

Two  broad  categories  of  forecasting  exist:  exploratory  and 
normative.  Exploratory  forecasting  techniques  consider  historical 
trends  and  extrapolate  into  the  future  to  see  what  may  happen.  The 


29  A  Methodology  for  Assessing  Acquisition  Technical  Risk,  Harrison,  William,  Gerald,  Jr.,  p. 

62. 

30  Forecasting  Technology  Uncertainty  in  Preliminary  Aircraft  Design,  Michelle  R.  Kirby  and 
Dimitri  N.  Mavris,  p.  11. 
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feasibility  of  this  process  depends  upon  an  assumption  that 
progress  is  evolutionary  and  does  follow  a  regular  pattern.  The 
normative  method  begins  with  future  goals  and  works  backward  to 
identify  the  levels  of  performance  needed  to  obtain  the  desired 
goals,  if  at  all  achievable  with  the  resources  available.  Either 
perspective  utilizes  one,  or  combinations,  of  four  traditional 
forecasting  techniques:  S-curves,  trend  extrapolation,  the  Delphi 
method,  or  scenario  development.  The  first  two  techniques  assume 
a  functional  form  of  previous  technological  growth  and  extrapolate 
to  a  future  time.  Again,  sufficient  information  must  exist  for  the 
forecast  to  be  accurate  and  of  value  to  the  decision-maker.  The 
Delphi  method  is  a  structured  means  of  incorporating  expert 
opinions  (usually  subjective)  through  questionnaires  and  controlled 
feedback  to  estimate  a  technology  impact  and  the  confidence  of 
achieving  that  impact.31 

It  is  also  true  in  the  case  of  US  programs  that,  as  in  the  Kirby  and  Mavris 
report,  that  TRL  data  is  often  derived  subjectively,  relying  on  the  subject  matter 
expert  opinion  of  program  managers,  using  a  method  like  the  Delphi  method. 

3.  Accuracy  OF  T-50  TRL  Assessment 

Based  on  the  report,  “Analytic  evaluation  on  technology  acquisition  of  T-50 
program”  published  by  the  Seoul  University  Aerodynamic  Institute,  the  T-50 
program  has  13  critical  technology  parts.  This  report  lists  these  critical 
technology  parts,  and  it  provides  some  conclusions. 

•  Eight  parts  (Aerodynamic,  Structure,  Detail  System,  Test  and 
Evaluation,  Cockpit,  Design  of  Shape,  Summary  of  System, 
Training  System)  have  made  fairly  good  progress  and  could 
achieve  the  expected  goal  until  the  time  of  weapon  research  and 
development 

•  Four  parts  (Avionics,  Prototype  Manufacturing,  Flight  control, 
Thrust  system)  have  not  enough  technology  and  especially  avionic 
parts  need  continuing  effort  because  it  has  a  little  chance  of 
achieving  the  goal.  Even  though  flight  control  part  does  not  make 
good  progress,  the  ongoing  effort  will  achieve  the  goal. 

•  Supply  and  support  system  does  not  have  enough  technology 
because  of  a  late  start. 


31  Forecasting  Technology  Uncertainty  in  Preliminary  Aircraft  Design,  Michelle  R.  Kirby  and 
Dimitri  N.  Mavris,  p.  4. 
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Number 

Part 

Significance(%) 

TRL 

Achieved 

Level 

1 

Aerodynamic 

8 

7.9 

88% 

2 

Structure 

8 

7.5 

83% 

3 

Flight  Control 

18 

4.9 

54% 

4 

Avionics 

15 

4.8 

53% 

5 

Detail  System 

5 

6.9 

77% 

6 

Prototype  Manufacturing 

5 

7.1 

79% 

7 

Test  and  Evaluation 

8 

6.9 

77% 

8 

Cockpit 

5 

6.0 

67% 

9 

Thrust  System 

5 

6.0 

67% 

10 

Design  of  Shape 

5 

8.6 

96% 

11 

Summary  of  System 

8 

7.6 

84% 

12 

Supply  and  Support 

System 

5 

5.7 

63% 

13 

Training  System 

5 

6.7 

74% 

Summary 

100 

m 

70% 

Table  7.  T-50  TRL  Assessment 

4.  US  Program  TRL  Assessment 

•  B-2  RMP:  the  program  reported  having  two  critical  technologies, 
but  a  formal  technology  readiness  assessment  conducted  in 
February  2004  concluded  that  two  additional  technologies  should 
be  considered  critical.  The  additional  two  technologies,  the 
receiver/exciter  for  the  electronic  driver  cards  and  aspects  of  the 
antenna  designed  to  help  keep  the  B-2’s  radar  signature  low,  are 
not  considered  fully  mature  but  are  approaching  maturity.32 

•  C-130  AMP:  Five  of  the  C-130  AMP’s  six  critical  technologies  are 
fully  mature,  as  the  program  is  primarily  utilizing  proven  commercial 
and  modified  off-the-shelf  technology  for  all  AMP  capabilities.  The 
remaining  critical  technology,  the  Terrain  Following  and  Terrain 
Avoidance  (TF/TA)  capability,  was  demonstrated  through  the  Air 
Force  Research  Lab’s  Quiet  Knight  advanced  technology 
demonstration.  33  The  program  manager  estimate  that  the 
technologies  that  make  up  the  Improved  TF/TA  System  were 
assessed  as  having  a  TRL  of  6  at  the  time  they  were  transitioned 
from  AFRL. 


32  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  32. 

33  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  34. 
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•  C-5AMP:  we  did  not  assess  the  C-5  AMP's  critical  technologies 
because  the  program  used  commercial  technologies  that  are 
considered  mature.  Program  officials  stated  that  those  technologies 
are  in  used  on  other  aircraft.34 

•  C-5  RERP:  The  program  manager  estimate  the  TRL  of  1 1  critical 
technologies  (Auxiliary  Power  Unit  (APU),  Engine  hardware,  Engine 
software,  Electrical  Power  System,  Elevator  Variable  Feel  Unit, 
Flight  Controls  System,  Hydraulic  Suction  Boost  Pump,  Integrated 
Diagnostic  System,  Nacelle/Thrust  Reverser,  Pylon,  Wing-to-Pylon 
Attachment  Fitting)  of  the  program. 

•  F-22:  The  F/A-22  did  not  have  mature  technology  at  the  start  of  the 
acquisition  program.  The  program  included  new  low-observable 
(stealth)  materials,  integrated  avionics,  and  propulsion  technology 
that  were  not  mature  at  this  time.  The  Air  Force  did  not  complete  an 
evaluation  of  stealth  technology  on  a  full-scale  model  of  the  aircraft 
until  several  years  into  development.  It  was  not  until  September 
2000,  or  9  years  into  development,  that  the  integrated  avionics 
reached  a  maturity  level  acceptable  to  begin  product  development. 
During  development,  the  integrated  avionics  was  a  source  of 
schedule  delays  and  cost  growth.35 

•  Global  Hawk:  The  program  manager  estimate  the  TRL  of  1 1  critical 
technologies  (Air  Vehicle  &  Engine,  Communications/Information 
Exchange,  Enhanced  Electro-optical/infrared  Sensors,  Enhanced 
Synthetic  Aperture  Radar,  Global  Air  Traffic  Management,  Lithium 
Batteries,  Multiple  Platform  -  Radar  Technology  Insertion  Program, 
Open  System  Architecture,  Signal  Intelligence  Phase  1, 2,  and  3) 

•  JSF:  GAO/NSIAD  -00-74  “joint  strike  fighter  acquisition”  report  the 
eight  key  technology  levels  at  the  start  of  development. 

•  JUCAS:  The  program  manager  estimate  the  TRL  of  six  critical 
technologies(Signature  Reduction  Technologies,  Advanced  Tactical 
Targeting  Technologies,  Secure  Robust  Communications,  Force 
Integration,  Interoperability,  and  Global  Information  Grid  (GIG) 
Compatibility,  Adaptive  Autonomous  Operations,  Operations  in 
Carrier  Controlled  Airspace) 

•  Predator:  Three  of  the  predator  B's  four  critical  technologies,  the 
synthetic  aperture  radar,  the  multispectral  targeting  system,  and  the 
air  vehicle,  are  fully  mature.36 


34  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  36. 

35  GAO-03-645T,  Best  Practice,  p.  1 1 . 

36  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  102. 
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•  V- 22  :  The  program  manager  estimated  that  the  results  of  testing 
the  MV-22  Block  A  under  the  expected  range  of  environmental 
conditions  in  which  it  is  expected  to  operate  was  successfully 
completed.  Assessment  of  whether  it  will  meet  its  operational 
requirements  was  made.  Problems  were  identified  and  corrective 
plans  were  developed  and  are  being  implemented  to  insure  a 
successful  Operational  Evaluation 

5.  Analysis  of  the  TRL 

Our  best  estimate  of  TRL  at  the  program  development  stage  is  based 
upon  input  from  the  program  manager.  We  have  some  data  that  indicates  TRL 
growth  form  stage-to-stage.  From  these  data,  we  have  observed  the  following 

•  The  growth  rate  from  development  to  design  review  stage  is  27%. 

•  The  growth  rate  the  one  from  design  to  development  stage  is  12%. 

These  growth  rates  imply  that  in  order  to  achieve  the  GAO  benchmark  of 

TRL  7  at  the  production  decision  stage,  it  is  sufficient  to  achieve  a  TRL  of  at  least 
5  at  the  development  phase.  Based  on  this  fact,  we  provide  the  following 
forecasts: 

•  JUCAS,  JSF,  and  F-22  programs  may  not  achieve  a  desirable  TRL 
level  at  the  production  stage. 

•  T-50,  program  having  TRL  6.3  at  the  development  stage,  will 
achieve  the  goal  of  TRL  7. 

6.  Summary 

GAO  and  program  manager  assessed  the  TRL  on  10  US  major  weapon 
acquisition  programs.  Each  of  these  10  major  weapon  acquisition  programs  has 
some  similarity  to  the  T-50  program. 

GAO  has  stated  that  a  best  practice  is  to  have  a  TRL  of  7,  and  that  this 
level  indicates  that  the  program  will  progress  over  time  without  potential  cost  and 
schedule  growth. 

We  observed  the  following  facts  about  JSF  and  F-22  from  our  data  set 

•  Each  had  an  immature  technology  at  the  development  phase. 

•  Program  acquisition  unit  costs  were  116.5%  on  the  F-22,  and 
26.2%  on  the  JSF,  These  estimates  were  made  in  GAO-05-301. 

•  Even  though  GAO  recommends,  as  a  best  practice,  the  use  of  the 
TRL  approach  as  part  of  the  program  management  for  a  major 
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weapons  system  acquisition,  some  programs  proceeded  without 
achieving  these  levels  or  doing  this  analysis.  That  is,  these 
programs  started  without  considering  the  TRL  of  the  key 
technologies,  and  therefore  failed  to  lay  the  basis  for  reducing  cost 
increases  and  schedule  delays. 

On  the  other  hand,  DoD  does  not  always  agree  with  the  GAO  reports, 
GAO  05-271  and  GAO  05-301.  An  example  of  this  is  the  assessment  of  JSF 
technology  maturity.  Furthermore,  US  programs  do  not  always  follow  the  GAO 
guidance.  Development  decisions  for  three  programs,  F-22,  JSF,  and  JUCAS 
were  made  without  achieving  the  sufficient  TRLs  of  the  key  technologies. 

In  a  comparison  between  the  US  programs  and  the  T-50  program,  we  see 
that  the  T-50  also  started  without  fully  matured  TRL.  Of  course,  TRL  can  be 
developed  with  the  same  step  of  the  technical  development  in  science.  Some 
weapon  development  program  needs  their  specific  technology  part,  which  have 
little  chance  of  being  developed  in  science.  With  the  anticipation  of  the  TRL 
growth  rate,  we  could  estimate  that  the  T-50  program  starting  with  TRL  6.3  is 
going  to  achieve  the  GAO  benchmark  of  TRL  7. 

B.  DESIGN  DRAWINGS 

1.  T-50  Design  Release  Status 

As  integral  parts  of  decreasing  the  risk  of  cost  increases  prior  to 
production,  the  T-50  program  implemented  two  processes:  Concurrent 
Engineering  (CE)  and  Computer  Aided  Design  (CAD) 

•  Computer  Aided  Design  (CAD):  T-50  is  designed  100%  using 
CAD  software.  Specifically,  T-50  was  designed  using  Computer 
Aided  Three-dimensional  Interactive  Application  (CATIA),  which  is 
often  used  as  a  design  tool  in  the  aerospace  industries,  and  plays  a 
major  role  in  making  design  drawings,  and  which  had  to  be  made 
prior  to  the  manufacturing.  “Although  CATIA  is  a  robust  design  tool, 
it  did  not  satisfy  all  LMTAS  integrated  product  development  (IPD) 
needs.  Lockheed  Martin  Tactical  Aircraft  Systems  (LMTAS) 
integrates  several  design  tools  to  serve  company-specific  needs. 
For  example,  LMTAS  selected  Computer-Aided  Three-Dimensional 
Interface  Applications  (CATIA)  (a  solid  modeling  package)  as  the 
core  design  software.  LMTAS  has  identified  several  specific  design 
functions  not  available  with  commercial  software  and  has  chosen  to 
customize  or  build  software  to  serve  these  specific  needs. 
Computer  Mock-up  (COMOK)  provides  an  example  of  enhancing 
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commercially  available  CATIA  software  to  satisfy  LMTAS  specific 
needs.  The  dynamic  sharing  of  design  iterations  enables 
concurrent,  IPD.  The  COMOK  database  management  system  is  the 
technology  that  provides  centralized  access  to  multiple 
configurations  defined  geometrically  by  CATIA  solid  models.  As 
new  programs  are  initiated,  COMOK  is  used  in  place  of  metal 
mock-ups.  Since  it  is  available  at  the  start  of  the  design  process,  it 
improves  design  integration  and  quality.”37. 

•  Concurrent  Engineering  (CE)  :ln  addition  to  the  use  of  CATIA,  T- 
50  used  the  concept  of  Concurrent  Engineering  that  is  “a  business 
strategy  which  replaces  the  traditional  product  development 
process  with  one  in  which  tasks  are  done  in  parallel  and  there  is  an 
early  consideration  for  every  aspect  of  a  product's  development 
process.  This  strategy  focuses  on  the  optimization  and  distribution 
of  a  firm's  resources  in  the  design  and  development  process  to 
ensure  effective  and  efficient  product  development  process.”38 

For  the  T-50  program,  Table  8  provides  design  release  status  while  Figure 
8  shows  the  percentage  of  drawings  released  over  time  while  using  the  CATIA 
and  CE  processes  on  the  process  of  engineering  drawings. 


Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

total 

1999 

- 

- 

- 

“ 

- 

- 

- 

- 

9 

4 

67 

170 

250 

2000 

316 

489 

750 

974 

1504 

2,513 

1,443 

934 

931 

893 

654 

639 

12,040 

Total 

12,290 

Table  8.  T-50  Design  Drawing  Release 


37  Best  Manufacturing  Practices, 

http://www.bmpcoe.org/bestpractices/internal/lmtas/lmtas  3.html  (accessed  November  29,  2005) 

38  Concurrent  Engineering,  http://best.me.berkeley.edU/--pps/pps/concurrent.html#what 
(accessed  November  29,  2005) 
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Figure  8.  Percentage  of  Drawings  Released 

2.  US  Program  Design  Release  Status 

•  B-2  RMP:  The  program  has  completed  and  released  71  percent  of 
its  engineering  drawings  to  manufacturing.  The  program  office  has 
scheduled  the  design  readiness  review  for  June  2005  and  plans  to 
have  85  to  95  percent  of  its  drawings  released  by  that  time.39 

•  C-130  AMP:  The  program  office  has  made  progress  toward 
meeting  its  goal  of  releasing  90  percent  of  the  design  drawings  by 
design  readiness  review,  scheduled  for  August  2005.  This  will  be 
nine  months  sooner  than  anticipated  last  year,  due  to  the 
acceleration  of  key  program  dates  to  meet  Special  Operations 
Command  requirements.  Currently,  48  percent  of  the  design 
drawings  are  complete  and  could  be  released  to  manufacturing.40 

•  C-5AMP:  The  design  appears  stable  as  the  contractor  has  released 
100  percent  of  the  drawings  for  the  AMP.41 

•  C-5  RERP:  The  program  manager  estimate  C-5  RERP's  design  is 
stable.  3,250  out  of  total  3,300  were  completed  before  the 
December  2003  at  Critical  Design  Review. 

•  F-22  :  The  F/A-22  design  is  essentially  complete,  but  it  matured 
slowly,  taking  over  three  years  beyond  the  critical  design  review  to 
meet  best  practice  standards.  The  late  drawing  release  contributed 

39  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  32. 

40  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  34. 

41  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  36. 
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to  parts  shortages,  work  performed  out  of  sequence,  delayed  flight 
testing  and  increased  costs.  Design  changes  resulted  from  flight 
and  structural  tests.42 

•  Global  Hawk:  The  program  manager  defined  that  1 ,650  out  of  total 
1,800  were  completed  at  the  April  2004  Critical  Design  Review. 

•  JSF:  The  program  estimates  35  percent  of  the  engineering  drawing 
packages  are  expected  to  be  released  at  the  critical  design  review. 
Also,  prototype  testing  will  not  be  done  prior  to  the  design  review. 
The  design  will  not  be  stable  until  after  production  begins.43 

•  JUCAS:  The  program  manager  defined  that  2,400  out  of  total  2,800 
were  completed  at  the  September  2005  Critical  Design  Review. 

•  Predator:  The  program  office  expects  94  percent  of  the  expected 
increment  one  drawings  to  be  completed  by  the  April  2006  critical 
design  review,  which  has  been  delayed  about  seven  months. 
Program  officials  acknowledge  that  additional  drawings  will  be 
needed  for  subsequent  increments.44 

•  V- 22:  The  program  manager  defined  that  7,230  out  of  total  7,490 
were  completed  at  the  September  2002  Critical  Design  Review. 

Figure  9  shows  the  percentage  of  releasable  design  drawings  on  the  US 
programs  based  on  the  GAO  report  and  input  from  program  managers,  before 
CDR. 


42  GAO-02-701,  Best  Practices,  p.  25. 

43  GAO-05-271,  Joint  Strike  Fighter  Acquisition,  p.  17. 

44  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  101. 
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Figure  9.  Releasable  Drawings  before  CDR  of  US  programs 

3.  Proportion  of  Drawings  Released  at  CDR 

To  examine  design  stability,  we  reviewed  the  time  schedule  for 
transitioning  the  engineering  drawings  from  the  engineering  side  of  the  house  to 
the  manufacturing  side  of  the  house.  There  is  a  natural  need  to  demonstrate, 
before  beginning  initial  manufacturing,  that  the  design  of  the  product  meets  the 
stated  requirements  of  the  weapons  acquisition  program.  To  determine  the 
design  stability,  and  to  judge  whether  the  product  is  ready  for  the  next  phase  of 
development,  CDR  begins  to  review  the  product  from  the  perspective  of  the 
product’s  component,  then  continues  its  review  at  the  subsystem  level,  and, 
finally,  at  the  integrated,  system  level. 

GAO’s  best  practice  benchmark  for  design  stability  requires  at  least  90% 
of  the  engineering  drawings  to  be  completed  prior  to  Critical  Design  Review 
(CDR).  The  time  schedule  for  giving  the  engineering  drawing  from  engineering  to 
manufacturing  is  used  to  examine  design  stability,  and  we  developed  and 
analyzed,  as  a  measure  of  effectiveness,  the  percent  achievement  of  the  number 
of  design  drawings  which  are  releasable  to  the  manufacturing  of  each  program. 
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The  data  of  C-5RERP,  Global  Hawk,  JUCAS(X-47),  V-22  and  T-50  were 
estimated  and  provided  by  each  program  manager.  On  the  other  hand,  data  for 
the  other  programs  are  based  on  the  GAO  report,  which  had  already  collected 
and  judged  the  achievement.  The  GAO  report  also  shows  the  achievement 
before  CDR  of  the  B-2RMP  and  C-130AMP  programs.  The  Global  Hawk 
program  manager  also  provided  the  specific  drawing  data  at  the  start  of  the 
program.  The  column  of  the  expected  value  represents  the  expected 
achievement  from  the  program  start  to  the  CDR. 

The  table  below  displays  the  data  we  have  developed  on  engineering 
drawings. 


Aircraft 

Achievement 

Releasable 

Expectation 

Criteria 

before  CDR(%) 

drawings  at  CDR(%) 

(90%) 

B-2  RMP 

0.71 

0.90 

0.19 

0.9 

C-130  AMP 

0.48 

0.90 

0.42 

0.9 

C-5  AMP 

- 

1.00 

- 

0.9 

C-5  RERP 

- 

0.98 

- 

0.9 

F-22 

0.26 

- 

0.9 

Global  Hawk 

0.90 

0.92 

0.02 

0.9 

JSF 

- 

0.35 

- 

0.9 

JUCAS 

- 

0.86 

- 

0.9 

Predator 

- 

0.94 

- 

0.9 

V-22 

- 

0.97 

- 

0.9 

T-50 

- 

0.67 

- 

0.9 

Table  9.  Design  Drawing  Release  Status 

The  following  Figures  10  and  11  illustrate  that  the  Matured  Key  technologies 
make  the  design  stability  keep  going  well  during  the  weapon  acquisition  process. 
The  seven  programs,  B-2  RMP,  C-130  AMP,  C-5  AMP,  C-5  RERP,  Global  Hawk, 
Predator,  and  V-22,  which  have  at  least  five  at  the  development  phase,  have  a 
satisfactory  level  of  design  drawings  at  CDR.  Even  if  six  critical  technologies  of 
the  JUCAS  were  not  matured  at  the  development  phase,  the  achievement  of 
design  drawing  is  ready  to  enter  the  stability  level.  The  releasable  drawings  of  T- 
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50  program  at  CDR  is  not  at  a  sufficient  level  to  assure  design  stability,  even 
though  it  starts  with  TRL  above  5.  Two  programs  having  the  lower  TRL 
achievement  at  development  phase  also  show  the  lower  achievement  of  design 
stability. 


Drawing  Release 


Figure  10.  Design  Drawing  Release  versus  Criteria 
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Figure  1 1 .  CDR  Drawings  /  Total  Drawings 
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The  following  figure  shows  the  expectation  of  the  design  drawings  from 
development  to  the  time  of  CDR.  As  we  see,  the  data  of  only  three  programs,  B- 
2RMP,  C-130AMP  and  Global  Hawk,  are  available  to  represent,  because  each 
program  is  not  managed  and  monitored  by  each  program  manager  according  to 
the  time  phase. 


Achievement  vs  expectation 


120%-, 


B-2RMP  C-130AMP  C-5  AMP  C-5  RERP  F-22  Global  JSF  JUCAS  Predator  V-22  T-50 

Hawk 


□  Achievement  before  CDR  □  Expectation 


Figure  12.  Achievement  versus  Expectation 

4.  R&D  Cost  versus  Number  of  Drawings 

The  R&D  cost  for  five  programs  except  T-50  is  on  the  basis  of  GAO  report 
and  R&D  cost  of  the  T-50  is  from  the  published  book  for  the  advertisement  of  T- 
50.  We  can  obtain  the  ratio  of  the  number  of  drawings  from  the  Critical  Design 
Review  (CDR)  for  all  programs  which  we  have  analyzed,  but  obtaining  the  exact 
data  on  the  number  of  drawings  is  hard  to  get  even  from  the  program  managers. 
The  number  of  drawings  of  each  program  in  the  table  below  comes  from  the 
program  managers.  To  judge  the  relation  between  the  R&D  cost  and  number  of 
drawings,  we  divided  the  R&D  cost  by  the  number  of  drawings  and  tried  to  find 
some  results.  All  R&D  costs  are  normalized  to  the  FY2005.  A  five  percent 
inflation  rate  is  used  to  normalize  the  T-50  weapon  development  program.  The 
table  below  display  the  R&D  cost  versus  the  number  of  drawings. 
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Program 

R&D  cost 

#  of  drawings 

R&D/#  of  drawings 

C-5  RERP 

$1,537.4 

3300 

0.47 

Global  Hawk 

$2,528.9 

1800 

1.40 

V- 22 

$10,723.7 

7490 

1.43 

T-50 

$2,790 

12290 

0.20 

JUCAS 

$4,042.0 

2800 

1.44 

■  (  FY05  dollars  in  millions) 

Table  10.  R&D  Cost  Data 


It  is  analytically  pleasing  when  three  numbers  are  almost  the  same,  as  is 
the  case  for  Global  Hawk,  JUCAS  and  V- 22.  It  seems  that  $1 .4  M  is  some  sort  of 
constant  for  new  aircraft.  We  do  not  know  what  causes  this  constancy  but 
perhaps  it  can  be  a  topic  for  further  research.  T-50,  which  does  not  fit  the  pattern, 
is  a  tactical  aircraft,  whereas  Global  hawk  and  JUCAS  are  Unmanned  Air 
Vehicle’s  (UAV) 


#Drawings 


Figure  13.  R&D  Cost/#  of  Drawings  versus  #  of  Drawings 
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5.  Regression  on  the  Engineering  Drawings 

The  figure  below  shows  the  regression  line  on  the  percent  achievement  of 
releasable  design  drawings  at  CDR  versus  R&D  cost. 

•  Unmanned  programs,  Predator,  Global  hawk  and  Jucas,  have  a 
linear  relationship  along  the  R&D  cost.  These  programs  are 
identified  with  circles  in  Figure  14. 

•  Advancing  program,  C-5AMP,  C-5RERP,  C-130AMP  and  B-2RMP, 
have  a  big  variation  of  achievement  of  releasable  design  drawings 
depending  on  each  program.  These  programs  are  identified  with  a 
square  in  Figure  14. 

•  V- 22  also  arrived  high  level  of  drawing  achievement  as  other 
advancing  programs  even  though  it  has  higher  R&D  costs  than 
other  advancing  programs. 

•  The  new  and  advanced  program,  F-22  and  JSF  with  substantial 
R&D  costs  arrived  a  lower  level  of  CDR  achievement.  These 
programs  are  identified  with  triangles  in  Figure  14. 

•  T-50  program,  which  is  a  new  program,  has  better  level  than  F-22 
and  JSF,  but  a  low  achievement  when  we  considered  the  low  R&D 
cost  programs. 


$0  $5,000  $10,000  $15,000  $20,000  $25,000  $30,000  $35,000  $40,000  $45,000 

R&D  cost 


Figure  14.  Percentage  Achievement  at  CDR  versus  R&D  Cost 

Figure  13  represents  the  percent  achievement  of  design  drawings  at  CDR 
versus  number  of  design  drawings  of  the  five  programs.  Our  efforts  to  find  the 
exact  number  of  design  drawings  had  trouble  in  judging  whether  the  program 
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manager  had  updated  the  number  of  each  program  during  the  whole 
development  period.  Therefore,  we  can  obtain  only  reliable  number  of  design 
drawings  on  the  five  programs,  Global  Hawk,  JUCAS,  C-5RERP,  v-22  and  T-50. 

•  The  only  regression  analysis  on  the  manned  programs,  C-5RERP, 
V-22  and  T-50  result  in  a  difficulty  of  finding  the  relationship 
between  percent  achievement  and  number  of  drawings  due  to  the 
little  data  information. 

•  This  regression  trend  shows  that  the  uncomplicated  programs  (that 
is,  those  having  fewer  design  drawings)  arrived  at  a  more 
satisfactory  level  of  releasable  design  drawings  than  the  programs 
having  more  design  drawings. 

•  T-50  has  more  design  drawings  than  advanced  program  and 
unmanned  program.  Its  achievement  of  design  drawing,  67  percent, 
at  CDR  is  below  the  regression  line.  If  the  T-50  also  achieved  the 
about  90  percent  of  drawings  at  CDR,  the  regression  line  would  be 
the  flat  line  and  will  have  very  little  slope  of  regression  line. 


Figure  15.  Percentage  Achievement  at  CDR  versus  #  Drawings 

6.  Summary 

The  US  GAO  had  set  the  benchmark  that  the  achievement  of  engineering 
drawing  stability  requires  at  least  90  percent  of  engineering  drawings  to  be 
releasable  by  CDR.  We  observed  the  following  from  our  dataset: 
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•  Even  thought  this  benchmark  gives  good  guidance  by  which  to 
judge  the  design  stability  of  a  major  weapon  acquisition  program, 
some  programs  do  not  follow  the  GAO  guidance 

•  The  seven  programs,  B-2  RMP,  C-130  AMP,  C-5  AMP,  C-5  RERP, 
Global  Hawk,  Predator,  and  V-22,  which  have  achieved  a  TRL 
score  of  at  least  5  at  the  development  phase,  also  meet  or  exceed 
GAO’s  90  percent  engineering  drawing  benchmark. 

•  Even  if  six  critical  technologies  of  the  JUCAS  are  not  matured  at 
the  development  phase,  the  program  has  achieved  the  design 
drawing  benchmark  and  is  stable  on  this  dimension. 

•  Two  programs,  F-22  and  JSF,  having  the  lower  TRL  achievement 
at  development  phase  also  show  the  lower  achievement  of  design 
stability. 

•  The  percent  of  achievement  is  correlated  with  program  complexity, 
where  complexity  is  either  research  and  development  cost  or 
number  of  design  drawings. 

The  relationship  between  the  achievement  of  releasable  design  drawings 
and  level  of  R&D  cost  shows  the  following: 

•  Unmanned  programs,  Predator,  Global  hawk  and  JUCAS,  have  a 
linear  relationship  along  the  R&D  cost. 

•  Advancing  program,  C-5AMP,  C-5RERP,  C-130AMP  and  B-2RMP, 
have  a  big  variation  of  achievement  of  releasable  design  drawings 
depending  on  each  program 

•  V-22  also  arrived  high  level  of  drawing  achievement  as  other 
advancing  programs  even  though  it  has  high  R&D  cost  than  other 
advancing  programs. 

•  The  new  and  advanced  program,  F-22  and  JSF,  having  a  very  high 
R&D  achieved  a  lower  level  of  CDR  achievement. 

We  investigated  the  relationship  between  the  achievement  of  releasable 
design  drawings  and  the  number  of  design  drawings  for  the  five  programs  for 
which  we  have  data.  The  observations  are: 

•  The  only  regression  analysis  on  the  three  data  points  which 
represent  manned  programs,  C-5RERP,  V-22  and  T-50,  has  poor 
statistics  and  we  concluded  that  it  is  difficult  to  find  a  linear 
relationship  between  percent  achievement  and  number  of  drawings. 
This  is  probably  due  to  the  little  data  information  available  to  us. 
This  regression  shows  that  the  less  complicated  programs  having 
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fewer  design  drawings  achieved  a  satisfactory  level  of  releasable 
design  drawings  than  the  programs  having  many  number  of  design 
drawings. 

We  considered  the  T-50  program  with  the  following  observations: 

•  T-50  TRLs  are  above  5 

•  T-50  achived  releasable  design  drawings  of  67  percent  at  CDR, 
which  is  below  GAO  guidance,  so  that  T-50  is  at  risk  for  not 
showing  design  stability. 

T-50  has  more  design  drawings  than  advanced  program  and  unmanned 
program  among  five  programs 

C.  PRODUCTION 

1.  T-50  SPC  Process  Plan 

This  paragraph  describes  the  T-50  SPC  plan  during  the  Full  Scale 
Development  and  Production  period  and  is  translated  from  the  T-50  SPC  plan. 

a.  Definition  of  SPC  of  T-50  Program 

Statistical  Process  Control  (SPC)  is  a  prevention  focused  process 
management  method  that  is  used  to  identify  sources  of  malfunctions  that  are 
occurred  by  inherent  variances,  materials,  workers,  management  methods, 
measuring  methods  and  working  environment,  through  quality  characteristic 
analyses  which  separate,  measure  and  quantify  causes  of  malfunctions  through 
statistical  methods. 

b.  Measuring,  Analyzing  and  Improving  Process 

SPC  plans  monitoring,  measuring,  analyzing  and  continuous 
improvement  process  and  execute  them  for  relevant  events  as  follows: 

•  Verification  of  product  suitability 

•  Ensuring  suitability  of  quality  management  systems,  and 

•  Continuous  improvements  of  effectiveness  of  quality  management 
systems.  Monitoring,  measuring,  analyzing  and  continuous 
improvement  process  also  include  decision  making  process  on 
methods  and  a  level  of  implement  of  SPC. 
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c.  Statistical  Method 

By  implementing  statistical  methods,  we  can  ensure  continuous 
and  stable  quality  of  products.  This  is  achieved  through  forecasting  potential 
problems  and  then  devising  countermeasures  to  prevent  these  forecasted 
problems. 

•  Analyze  collected  data  through  statistical  method  and  manage 
factors  that  affect  product  quality. 

•  A  whole  inspection  of  product  is  a  general  rule.  However,  when 
implementing  a  sample  inspection,  it  is  require  to  establish  a 
sample  inspection  plan  and  get  the  plan  approved  by  the 
government. 

•  During  prototype  production,  SPC  is  implemented  as  a  data 
collecting  stage.  Upon  full  scale  production,  KC’s  will  be  identified 
for  fully  implementing  SPC. 

2.  T-50  Statistical  Process  Control 

We  reviewed  information  from  the  T-50  program,  which  is  at  the  beginning 
of  the  full-scale  production  phase.  T-50  prototype  production  was  performed 
without  fully  developed  statistical  control.  Below  is  information  about  the  T-50 
Statistical  Process  Control  (SPC). 

•  SPC  (Statistical  Process  Control),  during  the  stage  of  Full  Scale 
Development  (FSD),  was  defined  as  a  phase  of  data  collection  that 
is  necessary  to  identify  Key  Characteristics  (KC’s)  for  the  mass 
production. 

•  SPC  should  be  fully  implemented  upon  identifying  KC’s  during 
mass  production.  However,  according  to  one  of  our  assembly 
engineers,  there  has  not  been  an  instance  where  SPC  had  been 
implemented  directly  for  each  component  of  T-50  model. 

•  However,  there  have  been  a  number  of  cases  where  SPC  had 
been  implemented  to  improve  the  production  process  electronically, 
(i.e.,  reducing  ‘lead  time’  through  process  improvement) 

3.  US  Program  Production  Knowledge 

The  third  dimension  of  GAO’s  knowledge-based  approach  to  evaluating 
weapons-systems  acquisition  is  Statistical  Process  Quality  Control  (SPQC).  We 
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examined  each  aspect  of  our  programs  and  found,  surprisingly,  that  most 
programs  are  not  doing  this.  The  following  table45  displays  the  results. 


Programs 

Description 

B-2  RMP 

“Production  maturity  metrics  are  planned  to  be  formulated  during 
development.  These  metrics,  which  may  or  may  not  include 
manufacturing  process  control  data,  are  planned  to  be  used  as 
measures  of  progress  toward  production  maturity  during  a 
production  readiness  review  prior  to  the  start  of  production  in 
February  2007”. 

C-130  AMP 

“Funding  reductions  in  fiscal  years  2003  and  2004  delayed  the  C- 
130  AMP’s  development  program,  which  resulted  in  a 
rescheduling  of  program  milestones  and  rebase  lining  of  the 
program.  The  design  review,  low-rate  initial  production,  and 
production  readiness  decisions  were  all  delayed.  While  program 
officials  stated  that  the  delay  in  schedule  would  provide  more  time 
to  resolve  issues  with  the  TF/TA  technology  and  software,  the 
delay  in  fielding  was  not  acceptable  to  the  Special  Operations”. 

C-5  AMP 

“We  could  not  assess  the  production  maturity  because  most 
components  are  readily  available  as  commercial-off-the-shelf 
items.  This  equipment  is  being  used  on  other  military  and 
commercial  aircraft.  In  addition,  the  C-5  AMP  is  incorporating 
many  other  off-the-shelf  systems  and  equipment,  such  as  the 
embedded  global  positioning  system,  the  inertial  navigation 
system,  and  the  multifunction  control  and  display  units.  To  ensure 
production  maturity,  the  program  office  is  collecting  data  regarding 
modification  kit  availability  and  the  installation  schedules.” 

C-5  RERP 

“We  did  not  assess  the  C-5  RERP’s  production  maturity  because 
the  Air  Force  is  buying  commercially  available  items.  However,  we 
expect  that  production  maturity  would  be  at  a  high  level  because 
the  engines  have  been  commercially  available  for  many  years.” 

F-22 

"The  program  office  stopped  collecting  process  control  information 
in  November  2000.  The  contractor  estimated  that  nearly  half  of  the 
key  processes  had  reached  a  marginal  level  of  control,  but  not  up 
to  best  practice  standards.  The  Air  Force  relies  on  the  contractor’s 
quality  system  to  verify  manufacturing  and  performance 
requirements  are  being  met.  However,  the  Air  Force  has  not 
demonstrated  the  F/A-22  can  achieve  its  reliability  goal  of  three 
hours  mean  time  between  maintenance.  It  does  not  expect  to 
achieve  this  goal  until  2008  when  most  of  the  aircraft  will  have 

45  GAO  05-301,  Assessments  of  Selected  Major  Weapon  Programs,  pp.  32,  34,  36,  38,  64, 
68,  80,  102,  118. 
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Programs 

Description 

already  been  bought.  Best  practices  call  for  meeting  reliability 
requirements  before  entering  production.  As  of  the  current 
production  estimate  by  approximately  mid-October  2004,  the  Air 
Force  had  only  demonstrated  about  22  percent  of  the  reliability 
required." 

Global 

Hawk 

“Although  production  experience  and  lessons  learned  on  the  RQ- 
4A  will  benefit  the  RQ-4B  program,  the  new  model  requires 
different  and  more  complex  manufacturing  processes  and  tooling 
than  the  original  model.  Officials  have  not  implemented,  and  do 
not  plan  to  implement,  a  comprehensive  statistical  process  control 
program  to  demonstrate  that  new  manufacturing  processes  are  in 
control  and  capable  of  meeting  cost,  schedule,  and  quality  targets. 
Officials  have  started  to  identify  critical  manufacturing  processes 
and  will  continue  to  collect  performance  data  such  as  defect  and 
rework  rates  to  measure  product  quality.  There  are  continuing 
concerns  about  the  quality  and  timeliness  of  several  key 
subcontractors,  which  negatively  affect  cost  and  schedule  of  both 
design  and  production  work.” 

JSF 

“The  program  office  is  collecting  information  on  the  JSF  production 
processes.  The  contractor  is  currently  in  the  process  of  identifying 
the  key  characteristics,  critical  manufacturing  processes  and 
capturing  some  early  data.  At  the  time  of  the  production  decision, 
the  program  will  not  have  demonstrated  that  the  aircraft  can  be 
produced  efficiently  or  with  expected  reliability.” 

JUCAS 

"The  J-UCAS  program  began  in  October  2003  with  technologies 
that  officials  project  will  sufficiently  mature  to  support  a  possible 
2010  start  of  operational  system  development,  so  there  is  no  data 
for  production.” 

Predator 

“Program  officials  said  the  contractor  does  not  plan  to  use 
statistical  process  controls  to  ensure  product  quality.  Instead,  they 
plan  to  use  other  quality  control  measures  such  as  scrap,  rework, 
and  repair  to  track  product  quality.  Also,  initial  operational  testing 
of  increment  one,  which  is  to  demonstrate  a  product  is  ready  for 
production,  is  not  scheduled  to  be  complete  until  September  2007. 
Testing  for  remaining  increments  has  not  been  determined.” 

V- 22 

“Process  management  is  becoming  more  robust  at  the  final 
assembly  site  on  each  major  fixture  assembly  using  Six  Sigma. 
Program  officials  point  to  the  delivery  of  aircraft  as  an  indication  of 
manufacturing  maturity.” 

Table  11.  US  Program  Production  Knowledge  (After:  GAO  05-301) 
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4.  Summary 

The  GAO  benchmark  suggests  a  standard  rule  for  describing  the 
production  maturity  and  for  estimating  if  the  programs  are  under  control  during 
the  weapon  production  phase.  As  the  GAO  states  it: 

To  determine  if  a  product’s  design  is  reliable  and  producible, 
successful  programs  use  statistical  process  control  to  bring 
manufacturing  processes  under  control  so  they  are  repeatable, 
sustainable,  and  consistently  producing  parts  within  quality 
standards.  The  collection  of  process  control  data  prior  to  a 
production  decision  can  enable  a  smooth  transition  from  product 
development  to  the  production  phase.46 

One  of  the  frequently  heard  comments  about  the  difficulty  of  implementing 
SPC  is  the  high  level  of  training  required  of  personnel  and  the  high  costs  of  these 
personnel.  Due  to  the  typical  shortage  and  high  cost  of  the  personnel  needed  to 
perform  such  tests,  we  focus  only  on  the  critical  processes  necessary  to  meet  the 
reliability  and  performance  standards.  GAO  used  the  Process  Capability  Index, 
which  can  be  used  to  compute  a  product  defect  rate  that  permits  us  to  assess 
such  critical  process  and  the  index  is  measured  the  application  of  statistical 
control. 

When  we  focus  on  the  critical  process,  the  SPC  data  needed  to  judge  a 
program’s  production  maturity  depends  on  the  particular  program  manager. 
Based  on  the  program  manager  data  and  some  reports,  most  programs  in  our 
dataset  do  not  follow  the  GAO’s  SPQC  guidance.  As  the  GAO  states:  “While 
the  absence  of  the  data  does  not  mean  that  production  processes  were  immature, 
it  does  prevent  an  assessment  against  an  objective  standard.’’47  On  the  other 
hand,  the  production  decision  of  some  programs  will  be  made  later,  and  some 
programs  have  a  plan  to  follow  the  SPC. 

Our  conclusions  on  the  programs  in  our  dataset  follow. 

•  C-130AMP,  JUCAS,  V- 22:  the  programs  are  under  development, 

so  it  is  too  early  to  judge  the  SPC. 

46  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  12. 

47  GAO-05-301,  Assessments  of  Selected  Major  Weapon  Programs,  p.  12. 
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•  C-5  AMP,  C-5  RERP:  production  maturity  seems  assured  since 
many  aspects  of  this  program  are  readily  available  as  commercial- 
off-the-shelf  items. 

•  B-2  RMP:  SPC  are  planned  to  be  formulated  later,  as  the  programs 
enter  their  production  phase. 

•  F-22,  Global  Hawk,  JSF,  Predator:  we  can  judge  that  the  programs 
do  not  follow  the  GAO  guidance 

•  T-50:  prototype  production  was  performed  without  fully  developed 
statistical  control  and  the  program  has  just  entered  the  full-scale 
production  phase.  While  SPC  should  be  fully  implemented  upon 
identifying  Key  Characteristic(KC’s)  during  mass  production,  our 
information  from  assembly  engineers  is  that  there  has  not  been  an 
instance  where  SPC  has  been  implemented  directly  for  any 
component  of  T-50  prototype  model.  On  the  other  hand,  there  have 
been  a  number  of  cases  where  SPC  had  been  implemented  to 
improve  the  production  time.  Looking  further  into  the  future,  T-50 
program  has  a  specific  plan  to  implement  SPC  during  the  Full  Scale 
Development  and  Production  period. 
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IV.  CONCLUSION 


A.  SUMMARY 

Technology  development  is  the  important  factor  in  estimating  whether  a 
weapons-development  program  could  meet  requirements  without  schedule 
delays  and  cost  overruns.  Acquisition-program  management  also  relies  on  the 
knowledge-based  approach  in  developing  new  weapon  programs  and  it  ensures 
achievement  of  key  results  through  high  levels  of  knowledge.  To  support 
program  management,  the  US  GAO  has  developed  a  methodology,  which 
implements  three  knowledge  points. 

•  Technology  maturity 

GAO  attempts  to  find  the  key  technologies,  which  are  measured  on 
a  scale  of  nine  levels  of  technological  maturity.  Starting  a  product 
development  with  a  technology  readiness  level  of  seven  is  a  GAO- 
identified  “best  practice”,  and  it  appears  to  ensure  the  satisfactory 
progress  of  weapon  acquisition  programs. 

•  Design  stability 

Successful  programs  obtain  design  stability  at  the  time  of  critical 

design  review  (CDR),  and  CDR  results  serve  as  criteria  for 

estimating  the  stability  of  engineering  drawings  released.  The 

benchmark  that  the  US  GAO  has  set  is  that  engineering-drawing 
stability  is  achieved  if  it  is  true  that  at  least  90  percent  of 
engineering  drawings  are  completed  (or  at  least  releasable)  before 
the  CDR.  ' 

•  Controlled  production  stability 

Products  manufactured  with  statistical  process  controls  in  place  are 
better  able  to  govern  costs  and  to  stay  within  schedule  than 

programs  that  do  not  have  SPC  in  place.  To  evaluate  such 

maturity,  the  components  of  statistical  process  control  are  used  and 
critical  manufacturing  processes  are  monitored  for  statistical  data. 
To  measure  the  performance  of  the  process,  a  process-capability 
index  (CPK)  is  used  to  determine  how  a  process  is  running 
compared  to  its  specification  limits  and  to  measure  production- 
process  capability  within  specified  limits.  The  CPK  index  below  1.33 
indicates  that  the  manufacturing  process  lacks  statistical  control 
and  acceptable  consistency. 

To  monitor  and  evaluate  the  programs’  maturity  and  stability,  we  collected 
data  from  some  US  programs,  B-2  RMP,  C-130  AMP,  C-5  AMP,  C-5  RERP,  F-22, 
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Global  Hawk,  JSF,  JUCAS,  Predator,  and  V- 22,  relevant  to  the  aircraft 
acquisition  and  advancing  program  to  determine  if  the  acquisition  also  follow  the 
GAO  guidance  or  not.  These  data  are  also  compared  to  the  T-50  acquisition 
program  data  to  estimate  if  the  program  would  make  good  progress  in  the 
research  and  development  phase. 

Interestingly,  even  though  the  US  GAO  had  been  trying  to  enhance  the 
military  capability  and  readiness,  some  acquisition  program  did  not  follow  the 
guidance  of  the  GAO.  The  following  table  shows  the  status  of  whether  each 
program  programs  in  our  dataset  follows  the  GAO  guidance  or  not.  In  this  table, 
we  followed  the  following  rules: 

•  A  TRL  score  above  7  is  considered  to  follow  the  GAO  guidance 

•  90  percent  achievement  of  engineering  drawings  at  CDR  was 
considered  to  follow  the  GAO  guidance. 


•  The  production  followed  by  the  SPC  is  considered  to  follow  the 
GAO  guidance  no  matter  what  the  CPK  is.  If  some  program  is 
under  development,  we  assume  that  SPQC  is  not  available. 


Programs 

TRL 

Engineering  Drawings 

SPQC 

B-2  RMP 

Yes 

Yes 

No 

C-130  AMP 

Yes 

Yes 

N/A 

C-5  AMP 

Yes 

Yes 

Yes 

C-5  RERP 

Yes 

Yes 

Yes 

F-22 

Yes 

No 

No 

Global  Hawk 

No 

Yes 

No 

JSF 

No 

No 

No 

JUCAS 

No 

No 

N/A 

Predator 

No 

Yes 

No 

V-22 

Yes 

Yes 

N/A 

T-50 

No 

No 

No 

Table  12.  Observance  of  GAO  Benchmarks 
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This  objective  of  this  thesis  is  to  investigate  if  the  GAO  knowledge  based- 
methodology  could  be  used  in  the  T-50  program  and,  if  possible,  to  judge  the 
progress  on  the  base  of  GAO  methodology  with  the  collected  data.  We  can  find 
some  conclusions  as  follows. 

•  We  can  find  the  technical  readiness  level,  releasable  design 
drawings  at  CDR  and  Statistical  Process  Control  plan  on  the  T-50 
program,  so  we  can  judge  the  T-50  program  on  the  basis  of  GAO 
report. 

•  Since  the  T-50  program  achieved  a  TRL  of  6.3  at  the  development 
stage,  we  believe  that  the  T-50  program  will  achieve  the  goal  of 
TRL  7. 

•  The  releasable  drawings  of  T-50  program  at  CDR  is  not  a  sufficient 
level  to  ensure  design  stability,  even  though  it  started  with  a  TRL 
above  5.  T-50  has  more  design  drawings  than  the  advanced 
programs  in  our  dataset  as  well  as  more  design  drawings  than  the 
unmanned  programs  in  our  dataset.  The  T-50  program’s 
achievement  of  design  drawing,  67  percent,  at  CDR  is  below  the 
“best  practice”  guidance  of  the  GAO. 

•  T-50  prototype  production  had  been  performed  without  fully 
developed  statistical  control  and  production  is  on  the  beginning 
phase  of  mass  production.  However,  there  has  not  been  an 
instance  where  SPC  has  been  implemented  directly  for  each 
component  of  T-50  prototype  model.  Fortunately,  T-50  program 
have  a  specific  plan  of  SPC  during  the  Full  Scale  Development  and 
Production  period. 

B.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

During  this  research,  we  identified  some  further  research  areas. 

The  first  is  to  forecast  the  maturation  of  technology  over  time  on  the  basis 
of  Kirby  and  Mavris  report  describing  the  impact  of  emerging  technology  on  the 
probabilistic  technology  maturation  process.  Even  though  this  report  focuses  on 
the  estimation  of  low  level  of  technology  maturity,  if  we  can  obtain  the  reliable 
TRL  of  weapon  program,  which  is  developed  with  TRL  above  5  from  the 
development  start  to  production,  we  can  compare  the  TRL  improvement  with 
estimation  of  the  report. 

The  second  is  to  find  the  relationship  between  the  number  of  drawings 
and  research  and  development  cost.  We  note  that  the  higher  number  of 
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designing  drawings,  the  higher  research  and  development  cost  for  the  programs 
in  our  dataset.  This  topic  needs  additional  data  to  explore  and  identify  the 
statistical  relationship  between  two  items. 

The  third  is  to  find  the  statistical  process  control  data  for  the  entire 
analysis  of  the  GAO  methodology.  The  environments  at  which  most  program 
managers  do  not  try  to  follow  the  GAO  guidance  make  it  hard  to  estimate  product 
stability.  Further  research  should  explore  the  SPC  and  CPK  data.  It  is  possible 
that  full-scale  analysis  of  GAO  methodology  would  also  yield  good  guidance  to 
the  new  developing  weapon  acquisition  program. 
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